*! version 0.0.8 25mar2024

Cross-Validation in Stata

Syntax

xv # [#] , metric(string asis)
[seed(integer)
uid(varlist)
tpoint(string asis) kfold(integer)
split(string asis)
results(string asis)
fitnm(string asis)
classes(integer) threshold(real)
pstub(string asis)
pmethod(string asis)
popts(string asis)
noall monitors(string asis)
display retain
valnm(string asis)
] : estimation command

Arguments
# The proportion of the data set to allocate to the training set.

[#] The proportion of the data set to allocate to the validation set.

Required
metric the name of a function from libxv or a user-defined function

Split
seed to set the pseudo-random number generator seed

uid a variable list for clustered sampling/splitting

tpoint a numeric, td(), tc(), or tC() value

kfold the number of K-Folds to create in the training set; default is kfold(1)

split a new variable name; default is split(_xvsplit)

Fit
results a stub for storing estimation results; default is results(xvres)

noall suppresses fitting the model to the entire training set

fitnm is used to name the collection storing the results; default is fitnm(xvfit).

Predict
pstub a new variable name for predicted values; default is pstub(_xvpred)

classes is used to specify the number of classes for classification models; default is classes(0).

threshold positive outcome threshold; default is threshold(0.5)

pmethod predicted statistic from predict

popts options passed to predict in addition to the method

noall suppresses prediction on entire training set for K-Fold cases; default is on

Validate
monitors zero or more function names from libxv or user-defined functions; default is

valnm is used to name the collection storing the results; default is valnm(xvval).

General
display display results in window; default is off

retain is an option used to preserve the stored estimation results, split variable, and predicted outcomes; default is off

Description

xv is a prefix command from the crossvalidate suite of tools to
implement cross-validation methods with Stata estimation commands. The xv
prefix can be used with arbitrary estimation commands that return results using
ereturn. It handles all four phases of cross-validation work: splitting
the data into training, validation, and/or test splits (see splitit);
fitting the model you specify with your estimation command (see fitit);
generating out-of-sample/held-out predicted values (see predictit); and
computing validation metrics and monitors (see validateit). xv is
a prefix that wraps the individual commands provided in the crossvalidate
suite intended to make the process of using cross-validation seemless and easy
for Stata users.

xv can be used to generate simple random sampling based train/test splits
or training/validation/test splits, K-Fold simple random samples, and Clustered
simple random sampling (panel data is a special case). A large number of
validation metrics are implemented in libxv and users are also free to
define their own mata functions that can be used by xv (see
libxv for additional information).

IMPORTANT: immediately following the xv command, you must specify
the proportion of the dataset to allocate to the training set. For k > 1, the
training proportion can be 1. If you specify a training set proportion of 1,
the noall option will be turned on automatically.

Options

Required

metric the name of a libxv or user-defined function, with the
function signature described in help libxv used to evaluate the fit
of the model on the held-out data. Only a single metric can be specified. For
user's who may be interested in hyperparameter tuning, this would be the value
that you would optimize with your hyperparameter tuning algorithm.

Splitting the Data

seed accepts integer values used to set the pseudo-random number
generator seed value.

uid accepts a variable list for clustered sampling/splitting. When an
argument is passed to this parameter entire clusters will be split into the
respective training and validation and/or training sets. When this option is
used with tpoint for xtset data, the panel variable must be nested
within the clusters defined by uid.

IMPORTANT!!! the order of the varlist passed to uid is
assumed to follow the hierarchy of the nesting in the data. Ensure that the
varlist passed to this option follows the same convention as used with
commands like mixed.

tpoint a time point delimiting the training split from it's corresponding
forecastting split. This can also be accomplished by passing the appropriate if
expression in your estimation command. Use of this option will result in an
additional variable with the suffix xv4 being created to identify the
forecasting set associated with each split/K-Fold. This is to ensure that
the forecasting period data will not affect the model training.

kfold is used to specify the number of K-Folds to create in the training
set. If the value is > 1, the training split proportion can be set to 1; if you
use a training set proportion of 1 with k > 1, the noall option will be
turned on for you automatically. For k == 1, or in the context of leave-one-out
cross-validation (see xvloo), you must provide a training set
proportion < 1.

split is used to specify the name of a new variable that will store the
identifiers for the splits in the data. If a value is passed to the split,
pstub, or results options it will trigger the retain option
to be turned on. The default value in the case where the retain option is
on and no value is passed to split is _xvsplit.

Model Fitting

results is used to estimates store the estimation
results from fitting the model on each fold in the dataset. When kfold is
> 1 and the noall option is omitted, an additional result will be stored
from fitting the model to the entire training split. Additionally, if a value
is passed to the split, pstub, or results options it will
trigger the retain option to be turned on. The default value in the case
where the retain option is on and no value is passed to results is
xvres.

fitnm is an option to pass a name to the collection created to store the
results. When is executed, it will initialize a new collection
or replace the existing collection with the same name. If you want to retain
the validation results from multiple executions, pass an argument to this
option. Note: this only affects users using Stata 17 or later.

noall is an option to prevent fitting, predicting, and validating a model
that is fitted to the entire training set when using K-Fold cross-validation
with a train/test or train/validation/test split. If the training set proportion
is set to 1, this option will be automatically enabled.

Predicting Out-of-Sample Results

pstub is used to define a new variable stub name for the predicted values
generated during the cross-validation process. When kfold > 1 and the
option noall is omitted, an additional variable is generated based on the
fit of the model to the entire training set, with the predictions made on the
validation set, if present, or the test set. Additionally, if a value
is passed to the split, pstub, or results options it will
trigger the retain option to be turned on. The default value in the case
where the retain option is on and no value is passed to pstub is
_xvpred. If this variable already exists, a suffix based on the timestamp when
the command is executed is added as a suffix.

classes is used to distinguish between models of non-categorical data (
classes(0)), binary data (classes(2)), and multinomial/ordinal
data (classes(>= 3)). You will only need to pass an argument to this
parameter if you are using some form of a classification model. Internally, it
is used to determine whether to call predict (in the case of
classes(0)) or classify (in all other cases).

threshold defines the probability cutoff used to determine a positive
classification for binary response models. This value functions the same way
as it does in the case of estat classification.

pmethod is passed internally to Stata's predict command to
generate the predicted values of the outcome for the out-of-sample data. When
the value of the classes option is 0, pmethod will default to xb.
If the value of classes is >= 2, pmethod will default to pr.

popts is passed internally to Stata's predict command to
generate the predicted values of the outcome for the out-of-sample data. For
multivariate outcome models, like sureg, this option can be used to
specify which of the equations should be used to predict the outcome of interest.
It can also be used to specify the nooffset option in single or
multi-equation models. Consult the predict documentation for the model
you are fitting for more details.

noall is an option to prevent fitting, predicting, and validating a model
that is fitted to the entire training set when using K-Fold cross-validation
with a train/test or train/validation/test split. If the training set proportion
is set to 1, this option will be automatically enabled.

Validating the Model

monitors the name of zero or more libxv or user-defined functions,
with the function signature described in help libxv used to
evaluate the fit of the model on the held-out data. These should not be used
when attempting to tune hyper parameters, but can still provide useful
information regarding the model fit characteristics.

valnm is an option to pass a name to the collection created to store the
results. When is executed, it will initialize a new collection
or replace the existing collection with the same name. If you want to retain
the validation results from multiple executions, pass an argument to this
option. Note: this only affects users using Stata 17 or later.

General Options

display an option to display the metric and monitor values in the results
window.

retain is used to retain the variables created, stored estimation results,
and dataset characteristics that are generated by xv. If an argument is
passed to either the split, pstub, or results options, retain
is automatically turned on and default names will be used for the split
variable, predicted outcome variable, and/or estimation results names if they
are not provided by the user.

Examples

Linear Regression Example

sysuse auto.dta, clear

xv 0.8, metric(msle) pstub(p) display retain monitors(mse rmse mae bias mbe ///

r2 mape smape rmsle rpd iic ccc huber phl rpiq r2ss) split(ivsplit): ///

reg price mpg i.foreign, vce(rob)

Poisson Example

webuse epilepsy.dta, clear

xv 0.6, metric(pll) pstub(p) display retain monitors(mse rmse mae bias mbe ///

r2 mape smape msle rmsle rpd iic ccc huber phl rpiq r2ss) split(poisplit) ///

pmethod(n): poisson seizures treat lbas lbas_trt lage v4

Instrumental Variables Example

webuse hsng2.dta, clear

xv 0.8, metric(msle) pstub(p) display retain monitors(mse rmse mae bias mbe ///

r2 mape smape rmsle rpd iic ccc huber phl rpiq r2ss) split(ivsplit): ///

ivregress 2sls rent pcturban (hsngval = faminc i.region), small

Mixed Effects Example

webuse nlswork.dta, clear

xv 0.8, metric(msle) pstub(p) uid(id) display retain split(mixsplit): ///

mixed ln_w grade age c.age#c.age ttl_exp tenure c.tenure#c.tenure || id:

Logit Example

webuse lbw.dta, clear

xv 0.6, metric(acc) pstub(p) display retain split(logsplit2) ///

monitors(sens recall spec prev ppv npv bacc mcc f1 jindex) classes(2): ///

logit low age lwt i.race smoke ptl ht ui

Multinomial Logit Example

webuse sysdsn1.dta, clear

xv 0.6, metric(mcacc) pstub(p) display classes(3) retain split(	csplit) ///

monitors(mcsens mcprec mcspec mcppv mcnpv mcbacc mcmcc mcf1 mcjindex ///

mcdetect mckappa): mlogit insure age male nonwhite i.site

Ordinal Logit Example

webuse fullauto.dta, clear

xv 0.6, metric(mcacc) pstub(p) display classes(5) retain split(mcsplit) ///

monitors(mcsens mcprec mcspec mcppv mcnpv mcbacc mcmcc mcf1 mcjindex ///

mcdetect mckappa): ologit rep77 foreign length mpg

Returned Values

The table below provides information about the macros returned by xv in
addition to the macros that are returned by the estimation command you specify.

Name

State Macros
e(rng) the current set rng setting

e(rngcurrent) the current RNG in effect

e(rngstate) the current state of the runiform() generator

e(rngseed) the seed last set for the stream RNG

e(rngstream) the current stream of the stream RNG

e(filename) the name of the file loaded in memory

e(filedate) the last saved date of the file in memory

e(version) the current Stata version

e(currentdate) the current date

e(currenttime) the current time

e(stflavor) the flavor of Stata currently in use (i.e., BE, SE, MP)

e(processors) the number of processors currently set for use

e(hostname) the name of the host machine

e(machinetype) description of the hardware platform

Splitting Macros
r(stype) the split method

r(flavor) the sampling method

r(splitter) the variable containing the sample split identifiers

r(forecastset) the variable containing the sample split identifiers for the forecast sample

r(training) the value(s) of the splitter variable that identify the training set(s)

r(validation) the value of the splitter variable that identifies the validation set

r(testing) the value of the splitter variable that identifies the test set

Fitting Macros
e(estres#) the name to store the estimation results on the #th fold.

e(estresnames) the names of all the estimation results

e(estresall) the name used to store the estimation results for the entire training set when K-Fold cross-validation is used.

e(predifin) the if expression to use when predicting on validation/test split.

e(kfpredifin) the if expression to use when predicting on the K-Fold hold out set.

Validation Scalars
r(metric#) contains the metric value for the corresponding K-Fold

r(`monitors'#) one scalar for each monitor passed to the monitors option, named by the monitor function with a numeric value to identify the corresponding K-Fold

r(metricall) contains the metric value for K-Fold CV fitted to all of the K-Folds

r(`monitors'all) contains the monitor values for K-Fold CV fitted to all of the K-Folds

Additional Information

If you have questions, comments, or find bugs, please submit an issue in the crossvalidate GitHub repository.

Contact

William R. Buchanan, Ph.D.

Sr. Research Scientist, SAG Corporation

SAG Corporation

wbuchanan at sagcorp [dot] com

Steven D. Brownell, Ph.D.

Economist, SAG Corporation

SAG Corporation

sbrownell at sagcorp [dot] com