*! version 0.0.3 01mar2024

Cross-Validation in Stata

Overview

The crossvalidate package includes several commands and a Mata library that
provide a range of possible cross-validation techniques that can be used with
any eclass Stata estimation command. For the majority of users,
the prefix commands (see xv or xvloo) should handle any of your
needs. On what we believe will be uncommon or rare occassions, a user made need
a bit more control over the process. In those cases, the lower level commands
provide a way for users to avoid programming the entire cross-validation process
while retaining the benefits that these commands provide.

IMPORTANT!!! If you intend to only use the lower-level commands, you will
need to call libxv first. This compiles the Mata source code into libxv
on your machine. If you are using either of the prefix commands xv or
xvloo, they will handle this step for you. However, if you intend to use
metric functions that you have defined prior to libxv compiling the mata
library, you should call libxv, then define your function, and then call
xv or xvloo. Prior to compiling libxv, the contents of
Mata are cleared to ensure that libxv only contains the functions that
should be included in the library.

This help file provides an overview of the commands included in the crossvalidate
package. We leave detailed information to the documentation for each of the
individual commands.

Commands

Command Name

Prefix Commands
xv Cross-Validation

xvloo Leave-One-Out Cross-Validation

Lower Level Commands
splitit Splits the dataset into train/test or train/validation/test splits

fitit Calls the estimation command on the appropriate split

predictit Predicts the outcome on the appropriate split

validateit Computes

Utility Commands
classify Used to manage

cmdmod Used for metaprogramming tasks in commands above

state Retrieves current settings and binds to the dataset

Prefix Commands

xv is a prefix command that should address the majority of use cases for
cross-validation. Use the prefix and provide the required arguments, then write
the estimation command you would use to fit your model under normal circumstances.
The command will handle spliting the data, fitting the model to the appropriate
subsets of data, generating the predicted values, and computing the quantities
of interest that describe the quality of the results. You can create simple
train/test and train/validation/test splits with or without K-Folds, using
simple random sampling or clustered sampling (including sampling of panel units).

xvloo is also a prefix command but is used to perform leave-one-out (LOO)
cross-validation. LOO can be though of as a special case of K-Fold
cross-validation where K is equal to the number of observations, or clusters, in
the training set; another way to think of this is using a jackknife for
cross-validation. Therefore, we strongly recommend only using this command when
working with smaller sample sizes. Additionally, if the number of observations
in your dataset plus the number of variables in the dataset plus 2 is greater
than the number of variables your version of Stata can support you will not be
able to use this prefix.

Lower Level Commands

splitit is a command called by the prefix commands to create the splits
in the data in memory. As mentioned above, you can create train/test and
train/validation/test splits with or without K-Folds, using simple random
sampling or clustered sampling (which includes sampling panel units). This
command generates a new variable to identify the splits in the dataset which is
required to be passed to the subsequent commands below.

fitit is a command called by the prefix commands to update and execute
the user supplied estimation command. The "update" made by this command is the
insertion, or modification, of an if expression that is used to ensure that the
estimation command you passed (either as an argument to this command or via the
prefix) is executed for the subset of data you intended. When used with K-Fold
cross-validation this command will also fit the model to the entire training set
in addition to each of the K-Folds, unless you tell it otherwise.

predictit is a command called by the prefix commands to manage and
generate the predicted values based on the previously fitted model. In the case
of K-Fold cross-validation, it ensures all the predicted values are stored in a
single variable with appropriate storage type (double precision for continuous
outcomes and byte for categorical outcomes). Like fitit, this command
will also generate predictions based on the model fitted to the entire training
set when using K-Fold cross-validation unless you tell it otherwise.

validateit is the last command called by the prefix commands and is used
to compute the validation/test metric of your choosing. We've included a
selection of metrics in the Mata library distributed with this package and they
are listed in the help file for validateit. Additionally, if there is a
validation metric that we have not implemented you may be able to use it by
defining a Mata function that follows our function signature requirements and
passing the name of that function to the appropriate option.

Utility Commands

classify is a utility called by the predictit command when fitting
classification models. This utility ensures that class identifiers are returned
as the predicted values for binomial, multinomial, and ordinal outcomes.

cmdmod is a utility called by fitit and possible predictit
to create the updated estimation command string and if expression for prediction.

state is a utility called by the xv and xvloo commands as
an option to bind information about the current state of the computer and
pseudo-random number generator if requested.

Additional Information

If you have questions, comments, or find bugs, please submit an issue in the crossvalidate GitHub repository.

Contact

William R. Buchanan, Ph.D.

Sr. Research Scientist, SAG Corporation

SAG Corporation

wbuchanan at sagcorp [dot] com

Steven D. Brownell, Ph.D.

Economist, SAG Corporation

SAG Corporation

sbrownell at sagcorp [dot] com