• At 2023 Conference a user requested capabilities to make cross-validation easier
• Limited cross-validation capabilities in the Stata ecosystem
• ML/AI techniques rely heavily on cross-validation
• The utility of inferential statistics is bound by their generalizability

What is Cross-Validation?

• A process used to evaluate out-of-sample performance of a model using existing data
• Cross-validation can be used to optimize the model's hyperparameters and/or model selection
• The process has four basic steps: split, fit, predict, validate/evaluate
• The process has either two or three phases based on the splitting technique: training, validation, and/or testing

All Cross-Validation Techniques are K-Fold Techniques!!!

TT Split w/ K = 1 Folds

TT Split w/ K > 1 Folds

TVT Split w/ K = 1 Folds

TVT Split w/ K > 1 Folds

Organization

• xv prefix for the majority of use cases
• xvloo prefix for leave-one-out cross-validation use cases
• Step-based Commands
• Utility Commands
• Mata Library with utility functions and validation metrics

Prefix Commands

					
xv|xvloo # [#], MEtric(string asis) [seed(integer) Uid(varlist)
TPoint(string asis) SPLit(string asis) KFold(integer)
RESults(string asis) fitnm(string asis) Classes(integer)
PStub(string asis) noall MOnitors(string asis) DISplay
RETain valnm(string asis) PMethod(string asis)
POpts(string asis)] : estimation command ...
				

Prefix Command Syntax example

					
sysuse auto.dta, clear
// 80/20 TT Split using MSE for the evaluation metric
xv 0.8, me(mse): reg price mpg
// 60/20/20 TVT split using MSE for the validation metric
xv 0.6 .2, me(mse): reg price mpg, vce(rob)
// 60/20/20 TVT 5-fold split using MSE for the validation metric
// and also reporting out Mean Avg % Error (MAPE)
xv 0.6 .2, me(mse) kf(5) mo(mape): reg price mpg, vce(rob)
				

Other Commands

• splitit, fitit, predictit, and validateit implement the individual steps of cross-validation
• classify ensures classification models return classes and not predicted probabilities
• cmdmod uses metaprogramming to adjust the command to apply to the appropriate subset of data
• state adds data characteristics and returns values about the state of the computing environment

splitit

• Requires a single proportion for TT splits or two proportions for TVT splits
• Can provide clustered splitting using the uid option
• Implements K-Folds with the kfold option and LOO with the loo option
• Stratification and panel/time-series are not implemented currently.

Libxv

• Includes parsing and string substitution functions
• Implements a cross-tabulation function returning a struct for classification methods
• A function to verify hierarchies when using clustered splitting
• Some statistical functions (e.g., dpois, kappawgts)
• And > 40 validation metrics for continuous, binary, and multiclass responses

User Supplied Metrics

					
real scalar metricName(string scalar pred, string scalar obs,
string scalar touse, | transmorphic matrix opts) {
real colvector y, yhat
real scalar metricValue
yhat = st_data(., pred, touse)
y = st_data(., obs, touse)
...
return(metricValue)
}
				
real scalar metricName(string scalar pred, string scalar obs,
string scalar touse, | transmorphic matrix opts) {
real colvector y, yhat
real scalar metricValue
yhat = st_data(., pred, touse)
y = st_data(., obs, touse)
...
return(metricValue)
}

Cross-validation for Continuous Outcomes

					
// Load an example dataset
sysuse auto.dta, clear
// Use an 80/20 TT split using MSE for the metric and also reporting out
// on several other model fit statistics for this simple regression model
xv .8, metric(mse) pstub(pred) display monitors(rmse((1, 2)) mae ///
		mape smape(("y"))): reg price mpg i.foreign
				
// Load an example dataset
sysuse auto.dta, clear
// Use an 80/20 TT split using MSE for the metric and also reporting out
// on several other model fit statistics for this simple regression model
xv .8, metric(mse) pstub(pred) display monitors(rmse((1, 2)) mae ///
		mape smape(("y"))): reg price mpg i.foreign
// Load an example dataset
sysuse auto.dta, clear
// Use an 80/20 TT split using MSE for the metric and also reporting out
// on several other model fit statistics for this simple regression model
xv .8, metric(mse) pstub(pred) display monitors(rmse((1, 2)) mae ///
		mape smape(("y"))): reg price mpg i.foreign

Cross-validation for Logit

					
// Load an example dataset
webuse lbw.dta, clear
// Use a 60/20/20 TVT split with 4-folds using ACC for the metric for logit model
xv 0.6 0.2, metric(acc) pstub(p) kfold(4) display split(logsplit2)	   ///
monitors(sens spec prev ppv npv bacc jindex) classes(2) seed(7779311): ///
logit low age lwt i.race smoke ptl ht ui
				
// Load an example dataset
webuse lbw.dta, clear
// Use a 60/20/20 TVT split with 4-folds using ACC for the metric for logit model
xv 0.6 0.2, metric(acc) pstub(p) kfold(4) display split(logsplit2)	   ///
monitors(sens spec prev ppv npv bacc jindex) classes(2) seed(7779311): ///
logit low age lwt i.race smoke ptl ht ui
// Load an example dataset
webuse lbw.dta, clear
// Use a 60/20/20 TVT split with 4-folds using ACC for the metric for logit model
xv 0.6 0.2, metric(acc) pstub(p) kfold(4) display split(logsplit2)	   ///
monitors(sens spec prev ppv npv bacc jindex) classes(2) seed(7779311): ///
logit low age lwt i.race smoke ptl ht ui

Cross-validation for Ordinal Outcomes

					
// Load an example dataset
webuse fullauto.dta, clear
// Drop observations 
set seed 111
drop if runiform() > 0.22
// Use a 80/20 TT split with n-folds for ordinal logit model
xvloo 0.8, metric(mcacc) pstub(p) display classes(5) retain ///
split(mcsplit) monitors(mcsens mcprec mcspec mcppv mcnpv mcbacc mcmcc mcf1  ///
mcjindex mcdetect mckappa): ologit rep77 foreign length mpg
				
// Load an example dataset
webuse fullauto.dta, clear
// Drop observations 
set seed 111
drop if runiform() > 0.22
// Use a 80/20 TT split with n-folds for ordinal logit model
xvloo 0.8, metric(mcacc) pstub(p) display classes(5) retain ///
split(mcsplit) monitors(mcsens mcprec mcspec mcppv mcnpv mcbacc mcmcc mcf1  ///
mcjindex mcdetect mckappa): ologit rep77 foreign length mpg
// Load an example dataset
webuse fullauto.dta, clear
// Drop observations 
set seed 111
drop if runiform() > 0.22
// Use a 80/20 TT split with n-folds for ordinal logit model
xvloo 0.8, metric(mcacc) pstub(p) display classes(5) retain ///
split(mcsplit) monitors(mcsens mcprec mcspec mcppv mcnpv mcbacc mcmcc mcf1  ///
mcjindex mcdetect mckappa): ologit rep77 foreign length mpg

Future work

• Implementing stratification and xt/ts appropriate methods in splitit
• Implementing hyperparameter tuning with grid search, simulated annealing, and/or "Bayesian" optimization.
• Extending metrics to provide micro and macro aggregation and metrics for other types of models

crossvalidate is easy to use, versatile, and extensible

Separation of concerns is not just for computer scientists!