*! version 0.0.6 22mar2024

Generating and Managing Model Predictions for Cross-Validation in Stata

Syntax

predictit "estimation command" , pstub(string asis)
[split(varname) classes(integer)
kfold(integer) threshold(real)
modifin(string asis) kfifin(string asis)
noall pmethod(string asis)
popts(string asis)]

Required
pstub a new variable name for predicted values

Optional
split name of the variable that identifies the training split(s)

classes is used to specify the number of classes for classification models; default is classes(0).

kfold specifies the number of folds in the training set; default is kfold(1).

threshold positive outcome threshold; default is threshold(0.5)

modifin a modified if expression

kfifin a modified if expression

noall suppresses prediction on entire training set for K-Fold cases

pmethod predicted statistic from predict; default is pmethod(pr)

popts options passed to predict in addition to the method

Description

Options

Required

pstub is used to define a new variable name/stub for the predicted values
from the validation/test set. When K-Fold cross-validation is used, this
option defines the name of the variable containing the predicted values from
each of the folds and will be used as a variable stub to store the results from
fitting the model to all of the training data.

Optional

split must contain the name of the variable that stores the test,
validation, and test splits. There will only be a single variable if the splits
were created using splitit. Additionally, if you are passing an
estimation command string as the first argument to this command, you must
pass the split variable name to this option.

classes is used to distinguish between models of non-categorical data (
classes(0)), binary data (classes(2)), and multinomial/ordinal
data (classes(>= 3)). You will only need to pass an argument to this
parameter if you are using some form of a classification model. Internally, it
is used to determine whether to call predict (in the case of
classes(0)) or classify (in all other cases).

kfold defines the number of K-Folds used for the training set. In other
places, we reference using K-Fold cross-validation in the more common form,
where the training set consists of multiple subsets of data. However, standard
train/test and train/validation/test splits are simply a special case of K-Fold
cross-validation where there is only a single fold.

threshold defines the probability cutoff used to determine a positive
classification for binary response models. This value functions the same way
as it does in the case of estat classification.

modifin is the if expression to use for the predictions on the individual
K-Folds. As a reminder, an 80/20 train/test split, is a special case of K-Fold
cross-validation with a single K-Fold. This option can be used in lieu of
passing the estimation command string. If an estimation command string is also
passed, the value passed to this parameter will be overwritten by the value
generated by the internal call to cmdmod. If fitit was called
prior to this command, you can pass `macval(e(predifin))' to this
parameter to provide the modified if expression for predictions.

kfifin is the if expression used to generate predictions on the entire
training set when using K-Fold cross-validation. This is typically used at the
conclusion of hyperparameter tuning to provide an assessment of the model fit
when fitted to all of the K-Folds simultaneously, prior to evaluating the
performance on a test set.

noall is an option to prevent predicting the outcome for a model fitted
to the entire training set when using K-Fold cross-validation. If this option
is used, kfifin will have no effect since the relevant predictions will
not be generated.

pmethod is passed internally to Stata's predict command to
generate the predicted values of the outcome for the out-of-sample data. The
default value used by predictit depends on the value passed to the
classes option. When option classes is set to 0 the prediction
method will default to xb; in all other instances, the prediction method
will default to pr.

popts is passed internally to Stata's predict command to
generate the predicted values of the outcome for the out-of-sample data. For
multivariate outcome models, like sureg, this option can be used to
specify which of the equations should be used to predict the outcome of interest.
It can also be used to specify the nooffset option in single or
multi-equation models. Consult the predict documentation for the model
you are fitting for more details.

Examples

Update these to reflect predictit

Load example data

webuse lbw, clear

Create a variable to identify the sample to fit the data to

g byte touse = runiformint(1, 6)

Fit a model to the data

fitit "logit low age smoke", spl(touse) kf(5) res(lmod))

Generate predictions for the five training folds

predictit, ps(pred) c(2) k(5) mod(`macval(r(predifin))')

Generate predicted values for the model fitted to the entire training set and the individual K-Folds

predictit, ps(pred) c(2) k(5) mod(`macval(r(predifin))') kfi(`macval(r(kfpredifin))')

Alternative syntax for the previous two examples

predictit "logit low age smoke", ps(pred) c(2) spl(touse) kf(5) noall

predictit "logit low age smoke", ps(pred) c(2) spl(touse) kf(5)

Additional Information

If you have questions, comments, or find bugs, please submit an issue in the crossvalidate GitHub repository.

Contact

William R. Buchanan, Ph.D.

Sr. Research Scientist, SAG Corporation

SAG Corporation

wbuchanan at sagcorp [dot] com

Steven D. Brownell, Ph.D.

Economist, SAG Corporation

SAG Corporation

sbrownell at sagcorp [dot] com