*! version 1.0.0 28Oct2020

Title

crossvalidate -- k-fold Crossvalidation

Syntax

crossvalidate newvar estimation_command depvar indepvars [if] [in] , [ folds(#) gen(newvar) shuffle options ]

crossv can be used as a synonym for crossvalidate

folds(crossvalidate##folds:#) Number of crossvalidation folds.

gen(newvar) Optionally, save the variable that splits observations into folds.

shuffle Optionally, put the data in random order.

options Additional options are passed to the estimation command

Description

crossvalidate computes k-fold cross-validated predictions from any Stata estimation command.
The command breaks a dataset into a number of subsets ("folds"), and for each
runs an estimator on everything but that subset, and predicts results.
crossvalidate stores predicted values in a
newly generated variable newvar.
Predicted values are generated by issuing the command predict newvar for each
fold and, depending on the estimation command, may represent probabilities, class predictions, or continuous
values.

crossvalidate passes whatever options you give it directly to the estimator; it handles only the folding.
Examples of estimation_command include svmachines and logistic.

Remarks

Only estimation commands that allow the use of predict after the estimation command can be used.
The program does not currently support the prediction of multiple variables as would be needed, for example,
for multinomial logistic regression.

Options

folds(#) Number of folds. Common values are 5 and 10. By default, folds(5) is used.

shuffle Optionally, generates random folds. This option uses random values; set the seed
if reproducibility is required.
By default, folds are in sort order.

gen(newvar) Optionally, save the variable that splits the observations into folds into new variable newvar.
The folds are labeled from 1, 2,...,<number of folds>.
This is useful to compute the average evaluation criterion for each fold later.

options Additional options are passed to the estimation command.

Examples

Typical classification with support vector machines:

. sysuse auto

. crossvalidate P svmachines foreign headroom gear_ratio weight, folds(5) type(svc) gamma(0.4) c(51)

. n err = foreign != P

. qui sum err

. di "Cross-validated error rate: `r(mean)'"

Nearest Neighbor classification:

. crossvalidate P discrim knn headroom gear_ratio weight, k(3) group(foreign)

Authors

Matthias Schonlau <schonlau@uwaterloo.ca>

Nick Guenther <nguenthe@uwaterloo.ca>