Help for crossfold

Description

crossfold performs k-fold cross-validation on a specified model in order to evaluate a model's ability to fit out-of-sample data.

This procedure splits the data randomly into k partitions, then for each partition it fits the specified model using the other k-1 groups and uses the resulting parameters to predict the dependent variable in the unused group.

Finally, crossfold reports a measure of goodness-of-fit from each attempt. The default evaluation metric is root mean squared error (RMSE).

Syntax

crossfold model [model_if] [model_in] [model_weight],
[eif()] [ein()] [eweight(varname)]
[stub(string)] [k(value)] [loud]
[mae] [r2]
[model_options]

Options

eif; ein Error evaluation if and in specifications place restrictions on the out-of-sample set that should be fit. Modelling if and in restrictions should be specified with the model.

eweight Weighting for error evaluation purposes. Model weights, identical or not, should be specified after the model.

stub() Specifies a stub name for naming estimation results and for the results matrix. The default is est.

k() Specifies a number of folds to carry out. The default is 5, and k cannot exceed 300 or the number of observations.

loud Displays each model as it is fit.

mae Calculates mean absolute errors (MAE) instead of RMSE.

r2 Calculates psuedo-R-squared (the square of the correlation coefficient of the predicted and actual values of the dependent variable) instead of RMSE.

model_options Modelling command options (such as fe for xtreg).

Examples

. sysuse nlsw88
(NLSW, 1988 extract)

. crossfold reg wage union

| RMSE
-------------+-----------
est1 | 4.171849
est2 | 4.105884
est3 | 4.038483
est4 | 4.151482
est5 | 4.171727

. crossfold reg wage union, mae

| MAE
-------------+-----------
est1 | 2.99209
est2 | 3.13541
est3 | 3.158161
est4 | 3.035878
est5 | 3.006016

.crossfold reg wage hours grade i.race i.industry i.occupation, r2

| Pseudo-R2
-------------+-----------
est1 | .2036234
est2 | .1804039
est3 | .2213548
est4 | .2159976
est5 | .1556564

. crossfold qreg wage union [weight=hours], eweight(hours) mae

(importance weights assumed)

| MAE
-------------+-----------
est1 | 3.078402
est2 | 2.864632
est3 | 2.846198
est4 | 2.989049
est5 | 2.990051

. crossfold qreg wage union collgrad age grade [weight=hours], eweight(hours) k(3) mae

(importance weights assumed)

| MAE
-------------+-----------
est1 | 2.449628
est2 | 2.700219
est3 | 2.588182

Saved Results

crossfold saves the model errors in the matrix r(stub) (which is named r(est) if no stub name is specified).

It also saves the model parameters under the names stub1 ... stubk. They can be recalled using estimates restore name.

Author

Benjamin Daniels
bbdaniels@gmail.com

References

Schonlau, Matthias. "Boosted regression (boosting): An intoductory tutorial and a Stata plugin." The Stata Journal (2005). 5, Number 3, pp.330-354.

FAQ: What are pseudo R-squareds? UCLA: Academic Technology Services, Statistical Consulting Group. http://www.ats.ucla.edu/stat/mult_pkg/faq/general/psuedo_rsquareds.htm (accessed February 14, 2012).