Help for loocv

Description

loocv - Leave-One-Out Cross-Validation.

For every observation in the estimating sample, loocv estimates the model specified by the user with all but the ith observation,
fits the model using the remaining N-1 observations and uses
the resulting parameters to predict the value of the dependent variable for the ith observation.
The prediction error for the ith variable is stored in memory and loocv proceeds to the next
observation in the sample.

loocv reports three goodness-of-fit measures: the root mean squared error (RMSE),
the mean absolute error (MAE), and the pseudo-R2 (the square of the correlation
coefficient of the predicted and observed values of the dependent variable).

Note that loocv estimates one regression for each observation in the estimating sample,
so it may take a while to run for large estimating samples.

Syntax

loocv model [if] [in] [weights], [eweights] [model_options]

Use

model The model one wishes to evaluate, such as "reg yvar x1var x2var"
(without the quotations).

weights Model weights. These weights are used to estimate the model, so
they must be compatible with the estimation method in use.

Options

eweights Weights for error evaluation purposes. These may be the same as or different
than the model weights, but must be specified. loocv does not assume that weights and
eweights are the same.

model_options Modelling command options (such as fe for xtreg).

Examples

. sysuse nlsw88
(NLSW, 1988 extract)

. loocv reg wage age collgrad married

Leave-One-Out Cross-Validation Results
-----------------------------------------
Method | Value
-------------------------+---------------
Root Mean Squared Errors | 5.5467815
Mean Absolute Errors | 3.3990273
Pseudo-R2 | .07084915
-----------------------------------------

. loocv reg wage hours grade i.race i.industry i.occupation

Leave-One-Out Cross-Validation Results
-----------------------------------------
Method | Value
-------------------------+---------------
Root Mean Squared Errors | 5.1635111
Mean Absolute Errors | 2.9866291
Pseudo-R2 | .20001635
-----------------------------------------

. loocv reg wage union [weight=hours], eweight(hours)

(analytic weights assumed)

Leave-One-Out Cross-Validation Results
-----------------------------------------
Method | Value
-------------------------+---------------
Root Mean Squared Errors | 4.1280683
Mean Absolute Errors | 3.0713142
Pseudo-R2 | .02064774
-----------------------------------------

. sort idcode

. loocv xi: ivreg2 wage collgrad (ttl_exp = age) in 1/100

Leave-One-Out Cross-Validation Results
-----------------------------------------
Method | Value
-------------------------+---------------
Root Mean Squared Errors | 7.4361463
Mean Absolute Errors | 5.2287439
Pseudo-R2 | .00001683
-----------------------------------------

. ret list

scalars:
r(r2) = .0000168265967266
r(mae) = 5.228743877410889
r(rmse) = 7.436146288561151

. mat list r(loocv)

r(loocv)[3,1]
LOOCV
RMSE 7.4361463
MAE 5.2287439
Pseudo-R2 .00001683

Saved Results

loocv returns the root mean squared error r(rmse), the mean
absolute deviation r(mae), and the pseudo R squared r(r2). It also stores
these results in the 3x1 matrix r(loocv).

Acknoledgments

loocv is based on Ben Daniels' crossfold module, that performs
x-fold cross-validation. To the extent possible, loocv's syntax and options
follow those in crossfold to facilitate its use among users familiar with
crossfold. Any errors are my own.

Author

Manuel Barron
manuel.barron@gmail.com

References

Daniels, Benjamin. 2012. "CROSSFOLD: Stata module to perform k-fold cross-validation"
(https://ideas.repec.org/c/boc/bocode/s457426.html)