*! version 0.0.8 22mar2024

Model Fitting for Cross-Validation in Stata

Syntax

fitit "estimation command" , split(passthru)
results(string asis) [ kfold(integer)
noall display name(string asis)]

Required
split name of the variable that identifies the training split(s)

results a stub for storing estimation results

Optional
kfold specifies the number of folds in the training set; default is kfold(1).

noall suppresses prediction on entire training set for K-Fold cases

display display results in window; default is off

name is used to name the collection storing the results; default is name(xvfit).

Description

Options

Required

split must contain the name of the variable that stores the test,
validation, and test splits. There will only be a single variable if the splits
were created using splitit.

results is used to estimates store the estimation
results from each of the kfold folds in the dataset. When used with
K-Fold cross-validation, the estimation results returned by ereturn will
be based on fitting the model to the entire training set. Results from each of
the training folds can be easily recovered using the appropriate reference
passed to the results option. In this case, you will need to add the
fold number as a suffix to the name you pass to the results option to
recover the estimation results for that fold. The fold number identifies the
held-out fold. So, the number 1 will recover the model that was fitted to all
of the training folds except number 1.

Optional

kfold defines the number of K-Folds used for the training set. In other
places, we reference using K-Fold cross-validation in the more common form,
where the training set consists of multiple subsets of data. However, standard
train/test and train/validation/test splits are simply a special case of K-Fold
cross-validation where there is only a single fold.

noall is an option to prevent predicting the outcome for a model fitted
to the entire training set when using K-Fold cross-validation. If this option
is used, kfifin will have no effect since the relevant predictions will
not be generated.

display an option to display the model fitting results in the result
window. If using a large number of K-Folds, it may be useful to not print all
of the model fitting results to the result window.

name is an option to pass a name to the collection created to store the
results. When is executed, it will initialize a new collection
or replace the existing collection with the same name. If you want to retain
the validation results from multiple executions, pass an argument to this
option. Note: this only affects users using Stata 17 or later.

Examples

Load example data

sysuse auto.dta, clear

Expand the data to create identical K-Folds

expand 6

Create a "split" identifier

bys make: g byte spvar = _n

Fit a model to each of the K-Folds and all of the training set

fitit "reg price mpg headroom", spl(spvar) res(tst) kf(5)

Fit the model only the the individual K-Folds

fitit "reg price mpg headroom", spl(spvar) res(tst) kf(5) noall

Returned Values

The following lists the names of the e-macros and their contents.

Name

e(estres#) the name to store the estimation results on the #th fold.

e(estresnames) the names of all the estimation results

e(estresall) the name used to store the estimation results for the entire training set when K-Fold cross-validation is used.

e(predifin) the if expression to use when predicting on validation/test split.

e(kfpredifin) the if expression to use when predicting on the K-Fold hold out set.

fitit also reposts the e-return values from the model fitted to the entire training set. As a reminder, when used with kfold > 1, the results returned by ereturn will come from fitting the model to the entire training set (e.g., all of the K-Folds simultaneously). The results from individual folds can be recovered by appending the held-out fold number to the value passed to results and calling estimates restore.

Additional Information

The play option in validateit is enabled by
collect preview. In addition to providing a convient way
for us to structure the display in a useful way, it also makes it easy for you -
the user - to export the validation results into any of several formats. The
results from fitit are all stored in the collection named
xvfit. For more information on how to export these results into the
format of your choosing, please see collect export.

If you have questions, comments, or find bugs, please submit an issue in the
crossvalidate GitHub repository.

Contact

William R. Buchanan, Ph.D.

Sr. Research Scientist, SAG Corporation

SAG Corporation

wbuchanan at sagcorp [dot] com

Steven D. Brownell, Ph.D.

Economist, SAG Corporation

SAG Corporation

sbrownell at sagcorp [dot] com