*! version 0.0.7 06mar2024

Dataset Splitting and Folding for Cross-Validation in Stata

Syntax

splitit # [#] [if] [in] [, uid(varlist)
tpoint(string asis) kfold(integer)
split(string asis) loo]

Arguments
# The proportion of the data set to allocate to the training set.

[#] The proportion of the data set to allocate to the validation set.

Options
uid a variable list for clustered sampling/splitting

tpoint a numeric, td(), tc(), or tC() value

kfold the number of K-Folds to create in the training set; default is kfold(1)

split a new variable name; default is split(_xvsplit)

loo is used only for Leave-One-Out cross-validation

Description

Options

uid accepts a variable list for clustered sampling/splitting. When an
argument is passed to this parameter entire clusters will be split into the
respective training and validation and/or training sets. When this option is
used with tpoint for xtset data, the panel variable must be nested
within the clusters defined by uid.

IMPORTANT!!! the order of the varlist passed to uid is
assumed to follow the hierarchy of the nesting in the data. Ensure that the
varlist passed to this option follows the same convention as used with
commands like mixed.

tpoint a time point delimiting the training split from it's corresponding
forecastting split. This can also be accomplished by passing the appropriate if
expression in your estimation command. Use of this option will result in an
additional variable with the suffix xv4 being created to identify the
forecasting set associated with each split/K-Fold. This is to ensure that
the forecasting period data will not affect the model training.

kfold is used to specify the number of K-Folds to create in the training
set.

split is used to specify the name of a new variable that will store the
identifiers for the splits in the data. If no argument is passed to this option
will store the result in a variable named _xvsplit.

loo is an option used to alter the underlying logic used to compute the
validation metrics/monitors. Since Leave-One-Out cross-validation is a special
case of K-Fold CV the model fitting and predictions are generated in a manner
consistent with all other K-Fold cases. However, with only a single unit in
each K-Fold in the LOO case computing validation metrics requires different
treatment. In the LOO case the validation metric/monitors are computed using
the predicted and observed values for all of the training set units in aggregate.

Examples

Load example dataset

sysuse auto, clear

Simple Random Sampling

Test/Train Split

splitit .8, ret(splitvar)

Train/Validation/Test Split

splitit .6 .2, ret(splitvar)

K-Fold Simple Random Sampling

Test/Train Split

splitit .8, ret(splitvar) k(5)

Train/Validation/Test Split

splitit .6 .2, ret(splitvar) k(5)

Clustered Random Sampling

Test/Train Split

splitit .8, ret(splitvar) uid(foreign)

Train/Validation/Test Split

splitit .6 .2, ret(splitvar) uid(foreign)

K-Fold Clustered Random Sampling

Test/Train Split

splitit .8, ret(splitvar) k(5) uid(foreign)

Train/Validation/Test Split

splitit .6 .2, ret(splitvar) k(5) uid(foreign)

Returned Values

The following lists the names of the r-macros and their contents.

r(stype) the split method

r(flavor) the sampling method

r(splitter) the variable containing the sample split identifiers

r(forecastset) the variable containing the sample split identifiers for the forecast sample

r(training) the value(s) of the splitter variable that identify the training set(s)

r(validation) the value of the splitter variable that identifies the validation set

r(testing) the value of the splitter variable that identifies the test set

Additional Information

If you have questions, comments, or find bugs, please submit an issue in the crossvalidate GitHub repository.

Contact

William R. Buchanan, Ph.D.

Sr. Research Scientist, SAG Corporation

SAG Corporation

wbuchanan at sagcorp [dot] com

Steven D. Brownell, Ph.D.

Economist, SAG Corporation

SAG Corporation

sbrownell at sagcorp [dot] com