{smcl}
{right:version 1.6.7 7.May.2022}
{title:}
{phang}
{cmd:cvauroc} {hline 2} Cross-validated Area Under the Curve for ROC Analysis after Predictive Modelling for Binary Outcomes
{title:Syntax}
{p 4 4 2}
{cmd: cvauroc} {depvar} {varlist} [if] [pw] [{cmd:,} Kfold() Seed() Probit Fit Detail graph graphlowess Detail]
{p_end}
{title:Description}
{p 4 4 2}
Receiver operating characteristic (ROC) analysis is used for comparing predictive models, both in model selection and model evaluation.
This method is often applied in clinical medicine and social science to assess the tradeoff between model sensitivity and specificity.
After fitting a binary logistic regression model with a set of independent variables, the predictive performance of this set of variables
- as assessed by the area under the curve (AUC) from a ROC curve - must be estimated for a sample (the 'test' sample) that is independent
of the sample used to predict the dependent variable (the 'training' sample). An important aspect of predictive modeling (regardless of
model type) is the ability of a model to generalize to new cases. Evaluating the predictive performance (AUC) of a set of independent
variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance.
K-fold cross-validation can be used to generate a more realistic estimate of predictive performance. To assess this ability in situations
in which the number of observations is not very large, {hi:cross-validation} and {hi:bootstrap} strategies are useful. {hi:cvauroc} is a
Stata rclass program that implements k-fold cross-validation for the AUC for a binary outcome after fitting a logit or probit regression model.
{hi:cvauroc} averages the AUCs corresponding to each fold and applies the bootstrap procedure to the cross-validated AUC to obtain statistical
inference and 95% bias corrected confidence intervals (CI). Furthermore, {hi:cvauroc} optionally provides the cross-validated fitted probabilities
for the dependent variable or outcome contianed in a new variable named {hi:_fit}, the sensitivity and specificity, contained in two new variables
named, {hi:_sen} and {hi:_spe}, and the plot for the mean cvAUC and k-fold ROC curves.
{title:Options}
{p 4 4 2}
{bf:pw} This option allows the user to include sampling weights (e.g. inverse-probability of censoring or treatment weights -IPCW or IPTW-).
{p_end}
{p 4 4 2}
{bf:Kfold} This option allows the user to set the number of random folds to an integer greater or equal than 0 (default = 10).
{p_end}
{p 4 4 2}
{bf:Seed} This option allows the user to set the random seed to an integer greater than 1 (default = 7777).
{p_end}
{p 4 4 2}
{bf:Probit} This option allows the user to fit a probit rather than a logit model (default).
{p_end}
{p 4 4 2}
{bf:Fit} This option allows the user to generate a new variable (_fit) containing the cross-validated probabilities for the dependent variable or outcome.
{p_end}
{p 4 4 2}
{bf:Detail} This option allows the user to tabulate the prevalence of the outcome, the sensitivity, specificity and false positive values by each level of the outcome fitted probabilities.
Furthermore, it creates two new variables containing the cross-validated sensitivity (_Sen) and specificity (_Spe) for the independent variable or predictor.
{p_end}
{p 4 4 2}
{bf:Graph} This option allows the user to graph the empirical cross-validated ROC curves for the respective k folds specified by the user.
{p_end}
{p 4 4 2}
{bf:Graphlowess} This option allows the user to graph a smoothed version of the mean cross-validated ROC curve and the empirical ROC curves for the respective k folds specified by the user.
{p_end}
{title:Example}
. use http://www.stata-press.com/data/r14/cattaneo2.dta
(Excerpt from Cattaneo (2010) Journal of Econometrics 155: 138-154)
. gen lbw = cond(bweight<2500,1,0.)
. cvauroc lbw mage medu mmarried prenatal fedu mbsmoke mrace order, kfold(10) seed(1972) probit fit det
1-fold (N=465).........AUC = 0.726
2-fold (N=464).........AUC = 0.752
3-fold (N=464).........AUC = 0.660
4-fold (N=464).........AUC = 0.621
5-fold (N=464).........AUC = 0.703
6-fold (N=465).........AUC = 0.742
7-fold (N=464).........AUC = 0.579
8-fold (N=464).........AUC = 0.641
9-fold (N=464).........AUC = 0.730
10-fold(N=464).........AUC = 0.704
Model:probit
Seed:1972
----------------------------------------------------------------
Cross-validated (cv) mean AUC, SD and Bootstrap Corrected 95%CI
----------------------------------------------------------------
cvMean AUC: | 0.6857
Booststrap corrected 95%CI: | 0.6348, 0.7079
cvSD AUC: | 0.0578
----------------------------------------------------------------
------------------------------------------------------------------
Mean cross-validated Sen, Spe and false(+) at lbw predicted values
------------------------------------------------------------------
Prevalence of lbw: 6.01%
------------------------
_Pp | _sen _spe _fp
-----+------------------------------
0.01 | 99.83 0.52 99.48
0.02 | 98.21 4.09 95.91
0.03 | 88.82 20.12 79.88
0.04 | 73.69 49.21 50.79
0.05 | 66.05 64.58 35.42
0.06 | 60.17 69.31 30.69
(output omitted ...)
0.39 | 0.27 99.98 0.02
------------------------------------------
*******************************************************
* Naive performance based on non-crossvalidated AUC *
*******************************************************
. logistic lbw mage medu mmarried prenatal fedu mbsmoke mrace order
Logistic regression Number of obs = 4,642
LR chi2(8) = 137.10
Prob > chi2 = 0.0000
Log likelihood = -986.35435 Pseudo R2 = 0.0650
------------------------------------------------------------------------------
lbw | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mage | .9959165 .0140441 -0.29 0.772 .9687674 1.023826
medu | .9451338 .0283732 -1.88 0.060 .8911276 1.002413
mmarried | .6109995 .1014788 -2.97 0.003 .4412328 .8460849
prenatal | .5886787 .073186 -4.26 0.000 .4613759 .7511069
fedu | 1.040936 .0214226 1.95 0.051 .9997838 1.083782
mbsmoke | 2.145619 .3055361 5.36 0.000 1.623086 2.836376
mrace | .3789501 .057913 -6.35 0.000 .2808648 .5112895
order | 1.05529 .0605811 0.94 0.349 .9429895 1.180964
_cons | .3468141 .1498299 -2.45 0.014 .1487176 .8087812
------------------------------------------------------------------------------
. predict fitted, pr
. roctab lbw fitted
ROC -Asymptotic Normal--
Obs Area Std. Err. [95% Conf. Interval]
------------------------------------------------------------
4,642 0.6939 0.0171 0.66041 0.72749
{title:Authors}
{p 4 4 2}
Miguel Angel Luque-Fernandez {break}
LSHTM, NCDE, Cancer Survival Group, London, UK {break}
Email: miguel-angel.luque@lshtm.ac.uk {break}
{p 4 4 2}
Camille Maringe {break}
LSHTM, NCDE, Cancer Survival Group, London, UK {break}
Email: camille.maringe at lshtm.ac.uk {break}
{p 4 4 2}
Daniel Redondo-Sanchez {break}
Biomedical Research Institute of Granada (ibs.Granada) {break}
Email: daniel.redondo.easp at juntadeandalucia.es {break}
{title:Acknowledgements}
{p 4 4 2}
Miguel Angel Luque Fernandez is supported by the Spanish National Institute of Health, Carlos III Miguel Servet I Investigator
Award (CP17/00206).
{title:References}
{p 4 4 2}
Luque-Fernandez MA, Redondo-Sánchez D, Maringe C. cvauroc: Command to compute cross-validated area under the curve for ROC analysis after predictive modeling for binary outcomes. The Stata Journal. 2019;19(3):615-625. doi:10.1177/1536867X19874237
{p_end}
{p 4 4 2}
Miguel Angel Luque-Fernandez (2016), Crossvalidation in Epidemiology {browse "http://scholar.harvard.edu/malf/presentations/cross-validation-epidemiology": Presentation}
{p_end}
{p 4 4 2}
StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP.
{p_end}
{p 4 4 2}
Hastie T., Tibshirani R., Friedman J., (2013). The elements of Statistical Learning, Data Mining, Inference and Prediction. Springer Series in Statistics.
{p_end}
{title:Also see}
{psee}
Online: {helpb crossfold} {helpb roctab} {helpb lsens} {helpb lroc} {helpb rocreg}
{p_end}