*! version 1.0 23 November 2024

[R] predperf -- Predictive performance

Syntax

Predictive performance versus 'true' observations (variable 1) of one single predictor (variable 2)

predperf varname1 varname2 [if] [in]
[weight]
[, options]

Predictive performance versus 'true' observations (variable 1) of two predictors (variables 2 and 3),
with comparison of their respective performances

predperf varname1 varname2 varname3 [if] [in]
[weight]
[, options]

metrics(#) type of performance indices to compute; default is 1 (see description)

rel equivalent to metrics(2): compute relative performance indices

log equivalent to metrics(3): compute logarithmic performance indices

level(cilevel) confidence level for the estimation of indices; default value is the confidence level set in STATA (usually 95)

df(# [#]) degrees of freedom to correct the computation of RMSE (or RMSPE or RMSLE); default is 0

sig display significance levels associated to performance indices

floor(#) constant to add to observations and predictions with metrics(3); default is 0

nograph omit the production of a graph illustrating the predictive performance

by, bysort and statsby are allowed; see prefix.

fweights and aweights are allowed; see weight.

Description

predperf computes two classical predictive performance indices resulting from the comparison of one or two predictors with corresponding 'true' observations: the first one describes the bias of the predictor,
i.e. its systematic departure from trueness, and the second one describes its imprecision,
i.e. the average error by which individual predictions randomly diverge from trueness.
The approach is directly inspired from Sheiner's and Beal's suggestions for measuring predictive performance (see references).
These simple, rudimentary indices are still widely used in many areas of applied sciences during the process of validation of predictive models with respect to true observations.

The indices are computed according to one of three possible types of metrics:

- by default, or if the option metrics(1) is specified, an additive approach is used to compute the Mean Error (ME) and the Root Mean Square Error (RMSE),
which are thus expressed in the units of the observations and predictions;
(RMSE is computed by STATA for most regression analyses to quantify the average lack-of-fit between predictions and observations;
it corresponds to Roy Wada's command rmse Yobs Ypred, raw)

ME = mean[Ypred - Yobs] and RMSE = sqrt(mean[(Ypred - Yobs)^2])

- specifying the option rel or metrics(2) will apply a relative approach in the computation of Mean Percent Error (MPE) and Root Mean Square Percent Error (RMSPE), both expressed in percentage ratios;

MPE = mean[(Ypred - Yobs)/Yobs] and RMSPE = sqrt(mean[(Ypred - Yobs)^2 / Yobs^2])

- specifying the option log or metrics(3) will apply a logarithmic approach in the computation of Mean Log Error (MLE), which will be expressed as geometric mean ratio,
while Root Mean Square Log Error (RMSLE) is expressed as a log-based coefficient of variation.

MLE = exp(mean[(Log(Ypred) - Log(Yobs))] and RMSLE = sqrt(mean[(Log(Ypred) - Log(Yobs))^2])

The calculated values of the indices are complemented by confidence intervals that give a clue on the precision of their estimation, under the assumption of independence and normality (or log-normality) of the prediction errors.
The sig option displays p-levels indicating whether ME or MPE depart significantly from 0, or of MLE from 1,
and whether RMSE, RMSPE or RMSLE differ significantly from the raw dispersion of observations (i.e. SD, CV or logarithmic SD),
i.e. whether the predictor predicts the observations better than purely random values with a similar distribution would.
In addition, if two predictors are evaluated, predperf will estimate the differences between respective bias and imprecision indices, and their significance (with sig).

Eventually, predperf produces a graph illustrating the prediction performance of the predictor(s), showing a scatterplot of observations versus predictions along the identity line (y=x),
complemented with prediction bands encompassing ±1 and ±2 RMSE (or RMSPE or RMSLE) and a line indicating the level of ME (or MPE or MLE).

Options

Main

metrics(#) specifies the type of performance indices to compute: (see description)

- 1 (default): absolute bias (ME) and imprecision (RMSE) in native observation units

- 2: relative bias (MPE) and imprecision (RMSPE), both as percentage ratios

- 3: relative bias (MLE) as geometric mean ratio, and relative imprecision (RMSLE) as logarithm

rel is equivalent to metrics(2), requesting the estimation of relative bias (MPE) and imprecision (RMSPE), both expressed as percentage ratios

log is equivalent to metrics(3), requesting the estimation of relative bias (MLE) as geometric mean ratio, and relative imprecision (RMSLE) as logarithm

level(cilevel) specifies the confidence level for the estimation of confidence intervals around estimated performance indices; the default value is the confidence level set in STATA (usually 95)

df(#) (or df(# #) if two predictors are compared) specifies the degrees of freedom for correcting the computation of RMSE (or RMSPE or RMSLE).
The default value is 0.
It is appropriate to leave the df value set to 0 as long as the predictions have been obtained from a model developed without reference to the observations used to validate its predictive performance.
Conversely, if the predictions have been calculated with a model based on the observations, e.g. by linear or non-lineaar regression,
it would by appropriate to set a df value equal to the number of estimated parameters of the model to calculate unbiased values for RMSE (or RMSPE or RMSLE).
When the option df(k) is specified, the following formula is used to correct RMSE calculation:

RMSE = sqrt(mean[(Ypred - Yobs)^2] * N/(N - k))

sig displays significance levels associated with the departure of ME or MPE from 0, or of MLE from 1,
and the difference of RMSE, RMSPE, or RMSLE from the raw dispersion of observations (i.e. respectively SD, SDP = percentage SD, or SDL = logarithmic SD).
If two predictors are evaluated, the significance of differences between respective bias and imprecision indices will be displayed as well.

floor(#) is used only with metrics(3). It specifies to add a constant value to observations and predictions before their Log-transormation, to make logarithmic transformation possible in case of negative or zero values.
The default floor value is 0.

nograph will omit the production of a graph illustrating the predictive performance.

Example

The dataset 'sheinerbealdata.dta' contains the data serving as example in the article of Sheiner and Beal (see references).

. use sheinerbealdata.dta

Evaluate the predictive performance of predictor A compared to observations Y, expressed as absolute bias (ME) and imprecision (RMSE):

. predperf y a

Evaluate the predictive performance of predictor A compared to observations Y, expressed as relative bias (MPE) and imprecision (RMSPE),
testing whether MPE differs significantly from zero and RMSPE from the CV of the set of observations:

. predperf y a, rel sig

Compare the predictive performances of predictors A and B versus observations Y, expressed as logarithmis bias (MLE) and imprecision (RMSLE), testing whether they differ significantly.
The lower limit of quantification of 0.1 for both predictors is used to replace zero values:

. predperf y a b, log sig floor(0.1)

Stored results

predperf stores the following values in r():

Scalars

r(N) number of pairs of observations-predictions included in the calculations

r(level) confidence level used in confidence interval calculations

r(df_1) degrees of freedom substracted fom N in RMSE/RSMPE/RSMLE calculation for predictor 1

r(me_1) ME/MPE/MLE estimate for predictor 1

r(me_lb_1) lower bound of the confidence interval of ME/MPE/MLE for predictor 1

r(me_ub_1) upper bound of the confidence interval of ME/MPE/MLE for predictor 1

r(p_me_1) significance level associated with ME/MPE/MLE for predictor 1

r(rmse_1) RMSE/RMSPE/RMSLE estimate for predictor 1

r(rmse_lb_1) lower bound of the confidence interval of RMSE/RMSPE/RMSLE for predictor 1

r(rmse_ub_1) upper bound of the confidence interval of RMSE/RMSPE/RMSLE for predictor 1

r(p_rmse_1) significance level associated with RMSE/RMSPE/RMSLE for predictor 1

Scalars computed for two predictors

r(df_2) degrees of freedom substracted fom N in RMSE/RSMPE/RSMLE calculation for predictor 2

r(me_2) ME/MPE/MLE estimate for predictor 2

r(me_lb_2) lower bound of the confidence interval of ME/MPE/MLE for predictor 2

r(me_ub_2) upper bound of the confidence interval of ME/MPE/MLE for predictor 2

r(p_me_2) significance level associated with ME/MPE/MLE for predictor 2

r(rmse_2) RMSE/RMSPE/RMSLE estimate for predictor 2

r(rmse_lb_2) lower bound of the confidence interval of RMSE/RMSPE/RMSLE for predictor 2

r(rmse_ub_2) upper bound of the confidence interval of RMSE/RMSPE/RMSLE for predictor 2

r(p_rmse_2 significance level associated with RMSE/RMSPE/RMSLE for predictor 2

r(me_12) difference of ME/MPE or ratio of MLE between predictors 1 and 2

r(me_lb_12) lower bound of the confidence interval of ME/MPE/MLE difference or ratio

r(me_ub_12) upper bound of the confidence interval of ME/MPE/MLE difference or ratio

r(p_me_12) significance level associated with the difference or ration of ME/MPE/MLE

r(rmse_12) difference of RMSE/RMSPE/RMSLE between predictors 1 and 2

r(rmse_lb_12) lower bound of the confidence interval of RMSE/RMSPE/RMSLE difference

r(rmse_ub_12) upper bound of the confidence interval of RMSE/RMSPE/RMSLE difference

r(p_rmse_12 significance level associated with the difference in RMSE/RMSPE/RMSLE

Author

Thierry Buclin, Clinical Pharmacology Service, University Hospital of Lausanne (CHUV) and Faculty of Biology and Medicine of the University of Lausanne (Switzerland) [e-mail address: Thierry.Buclin_at_chuv.ch]

References

Sheiner LB, Beal SL. Some suggestions for measuring predictive performance. Journal of Pharmacokinetics and Biopharmaceutics. 1981;9(4):503-12. DOI: 10.1007/BF01060893.