*! version 2.1.0 04Nov2020

*! version 2.0.0 06Sep2018

*! version 1.1.0 15Nov2017

*! version 1.0.1 10OOct2017

*! version 1.0.0 20June2015

Title

looclass -- Classification statistics for leave-one-out cross-validation of binary outcomes

Syntax

looclass depvar indepvars [if] [in] [weight]
, model(str) [cutoff(#) save figure model_options]

model(string) required. Choice of model used for classification includes "logit", "probit", "rforest", "boost", or "svmachines"

cutoff(#) positive outcome threshold used in classification tables; default is cutoff(0.5)

save saves observation-level predictions for full and test data

figure produces a graph presenting ROC curves for the full and test data

model_options allows all available options for the given model specified in the model() option

indepvars may contain factor variables in logit, probit and svmachines models; see fvvarlist.

fweights, iweights, and pweights are allowed for logit and probit models; see weight.

Description

looclass performs leave-one-out cross-validation for regression models with a binary outcome and then produces classification measures to assist in determining
the error rate (or conversely, the accuracy) of a prediction (classification) model.

Leave-one-out cross-validation is simply n-fold cross-validation, where n is the number of observations in the dataset. Each observation in turn is
left out, and the model is estimated for all remaining observations. The predicted value is then calculated for the one hold-out observation,
and the accuracy is determined as success or failure in predicting the outcome for that observation (see Chapter 5 in Witten et al. 2011). The results of all n
predictions are used to calculate the final error estimates (accuracy) displayed in the classification table and ROC curves generated by looclass.

Options

model(string) choice of model used for classification. The available models are "logit", "probit", "rforest",
"boost", and "svmachines" (the rforest, boost, and svmachines packages must
be downloaded for these modeling approaches to be available).

cutoff(#) specifies the value for determining whether an observation has a predicted
positive outcome in the classification tables. An observation is classified as positive if its
predicted probability is > #. The default is 0.5.

save generates two variables, "full" and "test", that contain the predictions estimated for the full and test data, respectively.

figure produces a graph presenting ROC curves for the full data and test data.

model_options specifies all available options for probit, logit, rforest, boost, and svmachines when the chosen model is
specified in the model() option.

Examples

Setup

. webuse lbw, clear

Leave-one-out cross validation using logistic regression and displaying the ROC curves

. looclass low age lwt i.race smoke ptl ht ui, model(logit) fig

Same as above but specifying rforest as the estimation model (note that "race" is not specified with the "i." prefix as above)

. looclass low age lwt race smoke ptl ht ui, model(rforest) fig

Same as the first example, but saving the predictions for the full and test data after restricting the sample to those <30 years

. looclass low age lwt i.race smoke ptl ht ui if age<30, model(logit) fig save

Specifying an svmachines model, and saving the predictions for the full and test data

. looclass low age lwt i.race smoke ptl ht ui, model(svmachines) fig save

Specifying a boosted regression model, and saving the predictions for the full and test data (note: boost is specified with maxiter = 100 to speed it up)

. looclass low age lwt race smoke ptl ht ui, model(boost) maxiter(100) fig save

Use cutpt (downloadable from SSC) to estimate the "optimal" cutpoint on both the full and test data

. cutpt low full, youden

. cutpt low test, youden

Stored results

looclass stores the following in r():

Scalars

r(P_corr_1) percent correctly classified - full

r(P_p1_1) sensitivity - full

r(P_n0_1) specificity - full

r(P_p0_1) false-positive rate given true negative - full

r(P_n1_1) false-negative rate given true positive - full

r(P_1p_1) positive predictive value - full

r(P_0n_1) negative predictive value - full

r(P_0p_1) false-positive rate given classified positive - full

r(P_1n_1) false-negative rate given classified negative - full

r(roc1) ROC curve - full

r(P_corr_2) percent correctly classified - test

r(P_p1_2) sensitivity - test

r(P_n0_2) specificity - test

r(P_p0_2) false-positive rate given true negative - test

r(P_n1_2) false-negative rate given true positive - test

r(P_1p_2) positive predictive value - test

r(P_0n_2) negative predictive value - test

r(P_0p_2) false-positive rate given classified positive - test

r(P_1n_2) false-negative rate given classified negative - test

r(roc2) ROC curve - test

r(rocp) p-value for chi-squared test between full and test ROC curves

References

Witten, I. H., E. Frank, and M. A. Hall. (2011) Data Mining: Practical Machine Learning Tools and Techniques (3 ed.). Boston: Elsevier Press.

Linden A. (2006) Measuring diagnostic and predictive accuracy in disease management: an introduction to receiver operating characteristic (ROC) analysis.
Journal of Evaluation in Clinical Practice 12: 132-139.

Acknowledgments

William McClain identified a bug that occured when using the [if][in] syntax due to not sorting the data before running the LOO loop.

Citation of looclass

looclass is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden, Ariel (2015). looclass: Stata module for generating classification statistics of Leave-One-Out cross-validation for binary outcomes.
http://ideas.repec.org/c/boc/bocode/s458032.html

Author

Ariel Linden

President, Linden Consulting Group, LLC

alinden@lindenconsulting.org

http://www.lindenconsulting.org

Also see

Online: probit, logit, estat classification, roccomp, rforest (if installed),
boost (if installed), svmachines (if installed), kfoldclass (if installed), cutpt (if installed)