*! version 2.2.1 04Nov2020

*! version 2.2.0 11Sep2018

*! version 2.0.0 25Aug2018

*! version 1.1.0 10Nov2017

*! version 1.0.0 06Oct2017

Title

kfoldclass -- Classification statistics for k-fold cross-validation of binary outcomes

Syntax

kfoldclass depvar indepvars [if] [in] [weight]
, model(str) [k(#) cutoff(#) save figure model_options]

model(string) required. Choice of model used for classification includes "logit", "probit", "rforest", "boost", or "svmachines"

k(#) number of folds (hold-out groups); default is 5

cutoff(#) positive outcome threshold used in classification tables; default is cutoff(0.5)

save saves observation-level predictions for full and test data

figure produces a graph presenting ROC curves for the full and test data

model_options allows all available options for the given model specified in the model() option

indepvars may contain factor variables in logit, probit and svmachines models; see fvvarlist.

fweights, iweights, and pweights are allowed for logit and probit models; see weight.

Description

kfoldclass performs k-fold cross-validation for models with a binary outcome and then produces classification measures to assist in determining
the error rate (or conversely, the accuracy) of a prediction (classification) model.

In k-fold cross-validation, each of the k hold-out groups in turn is left out, and the model is estimated for all remaining (k-1)
groups. The predicted value is then calculated for the one hold-out group, and the accuracy is determined as success or failure in predicting the outcome for that group
(see Chapter 5 in Witten et al. 2011). The results of all k predictions are used to calculate the final error estimates (accuracy) displayed
in the classification table and ROC curves generated by kfoldclass.

Remarks

In kfoldclass, the k-fold groups are generated in a manner that preserves the proportions of the dependent variable as in the full sample.
However, if the user sets k(#) too high, then it is possible that one or more k-fold groups will have no observations for one of the outcome levels.
This issue can be avoided by checking cell counts after running kfoldclass with the save option and then reviewing the cell distributions, using:

. tab group depvar, col

If any empty cells are noted within one or more groups, the user should reduce the number set in k(#) accordingly.

Options

model(string) choice of model used for classification. The available models are "logit", "probit", "rforest",
"boost", and "svmachines" (the rforest, boost, and svmachines packages must
be downloaded for these modeling approaches to be available).

k(#) specifies the number of k-folds (hold-out groups) to generate. The default is 5, and k cannot be less than 2 or greater than
the number of observations.

cutoff(#) specifies the value for determining whether an observation has a predicted
positive outcome in the classification tables. An observation is classified as positive if its
predicted probability is > #. The default is 0.5.

save generates three variables; "group" which is the integer representing the given it:k}-fold, and
"full" and "test", that contain the predictions estimated for the full and test data, respectively.

figure produces a graph presenting ROC curves for the full and test data.

model_options specifies all available options for probit, logit, rforest, boost, and svmachines when the chosen model is
specified in the model() option.

Examples

Setup

. webuse lbw, clear

k-fold cross validation using logistic regression with 5-folds and displaying the ROC curves

. kfoldclass low age lwt i.race smoke ptl ht ui, model(logit) k(5) fig

Same as above but specifying randomforest as the estimation model (note that "race" is not specified with the "i." prefix as above)

. kfoldclass low age lwt race smoke ptl ht ui, model(rforest) k(5) fig

Same as above but limiting age to less than 28

. kfoldclass low age lwt race smoke ptl ht ui if age<28, model(rforest) k(5) fig

Specifying an svmachines model, and saving the predictions for the full and test data

. kfoldclass low age lwt i.race smoke ptl ht ui, model(svmachines) k(5) fig save

Specifying a boosted regression model, and saving the predictions for the full and test data (note: boost generally runs extremely slow)

. kfoldclass low age lwt race smoke ptl ht ui, model(boost) k(5) fig save

After saving predictions in previous step, assessing whether the number of groups specified in k(#) is appropriate

. tab group low, col

Use cutpt (downloadable from SSC) to estimate the "optimal" cutpoint on both the full and test data

. cutpt low full, youden

. cutpt low test, youden

Stored results

kfoldclass stores the following in r():

Scalars

r(P_corr_1) percent correctly classified - full

r(P_p1_1) sensitivity - full

r(P_n0_1) specificity - full

r(P_p0_1) false-positive rate given true negative - full

r(P_n1_1) false-negative rate given true positive - full

r(P_1p_1) positive predictive value - full

r(P_0n_1) negative predictive value - full

r(P_0p_1) false-positive rate given classified positive - full

r(P_1n_1) false-negative rate given classified negative - full

r(roc1) ROC curve - full

r(P_corr_2) percent correctly classified - test

r(P_p1_2) sensitivity - test

r(P_n0_2) specificity - test

r(P_p0_2) false-positive rate given true negative - test

r(P_n1_2) false-negative rate given true positive - test

r(P_1p_2) positive predictive value - test

r(P_0n_2) negative predictive value - test

r(P_0p_2) false-positive rate given classified positive - test

r(P_1n_2) false-negative rate given classified negative - test

r(roc2) ROC curve - test

r(rocp) p-value for chi-squared test between full and test ROC curves

References

Witten, I. H., E. Frank, and M. A. Hall. (2011) Data Mining: Practical Machine Learning Tools and Techniques (3 ed.). Boston: Elsevier Press.

Linden, A. (2006) Measuring diagnostic and predictive accuracy in disease management: an introduction to receiver operating characteristic (ROC) analysis.
Journal of Evaluation in Clinical Practice 12: 132-139.

Citation of kfoldclass

kfoldclass is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden, Ariel (2017). kfoldclass: Stata module for generating classification statistics for k-fold cross-validation of binary outcomes.
http://ideas.repec.org/c/boc/bocode/s458127.html

Author

Ariel Linden

President, Linden Consulting Group, LLC

alinden@lindenconsulting.org

http://www.lindenconsulting.org

Acknowledgment

I wish to thank Nicholas J. Cox for his review of kfoldclass and suggestions for improvement. John Moran suggested the "save"
option.

Also see

Online: probit, logit, estat classification, roccomp, rforest (if installed),
boost (if installed), svmachines (if installed), looclass (if installed), cutpt (if installed)