version 1.6.7 7.May.2022

cvauroc -- Cross-validated Area Under the Curve for ROC Analysis after Predictive Modelling for Binary Outcomes

Syntax

cvauroc depvar varlist [if] [pw] [, Kfold() Seed() Probit Fit Detail graph graphlowess Detail]

Description

Receiver operating characteristic (ROC) analysis is used for comparing predictive models, both in model selection and model evaluation.
This method is often applied in clinical medicine and social science to assess the tradeoff between model sensitivity and specificity.
After fitting a binary logistic regression model with a set of independent variables, the predictive performance of this set of variables
- as assessed by the area under the curve (AUC) from a ROC curve - must be estimated for a sample (the 'test' sample) that is independent
of the sample used to predict the dependent variable (the 'training' sample). An important aspect of predictive modeling (regardless of
model type) is the ability of a model to generalize to new cases. Evaluating the predictive performance (AUC) of a set of independent
variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance.
K-fold cross-validation can be used to generate a more realistic estimate of predictive performance. To assess this ability in situations
in which the number of observations is not very large, cross-validation and bootstrap strategies are useful. cvauroc is a
Stata rclass program that implements k-fold cross-validation for the AUC for a binary outcome after fitting a logit or probit regression model.
cvauroc averages the AUCs corresponding to each fold and applies the bootstrap procedure to the cross-validated AUC to obtain statistical
inference and 95% bias corrected confidence intervals (CI). Furthermore, cvauroc optionally provides the cross-validated fitted probabilities
for the dependent variable or outcome contianed in a new variable named _fit, the sensitivity and specificity, contained in two new variables
named, _sen and _spe, and the plot for the mean cvAUC and k-fold ROC curves.

Options

pw This option allows the user to include sampling weights (e.g. inverse-probability of censoring or treatment weights -IPCW or IPTW-).

Kfold This option allows the user to set the number of random folds to an integer greater or equal than 0 (default = 10).

Seed This option allows the user to set the random seed to an integer greater than 1 (default = 7777).

Probit This option allows the user to fit a probit rather than a logit model (default).

Fit This option allows the user to generate a new variable (_fit) containing the cross-validated probabilities for the dependent variable or outcome.

Detail This option allows the user to tabulate the prevalence of the outcome, the sensitivity, specificity and false positive values by each level of the outcome fitted probabilities.
Furthermore, it creates two new variables containing the cross-validated sensitivity (_Sen) and specificity (_Spe) for the independent variable or predictor.

Graph This option allows the user to graph the empirical cross-validated ROC curves for the respective k folds specified by the user.

Graphlowess This option allows the user to graph a smoothed version of the mean cross-validated ROC curve and the empirical ROC curves for the respective k folds specified by the user.

Example

. use http://www.stata-press.com/data/r14/cattaneo2.dta
(Excerpt from Cattaneo (2010) Journal of Econometrics 155: 138-154)

. gen lbw = cond(bweight<2500,1,0.)

. cvauroc lbw mage medu mmarried prenatal fedu mbsmoke mrace order, kfold(10) seed(1972) probit fit det

1-fold (N=465).........AUC = 0.726
2-fold (N=464).........AUC = 0.752
3-fold (N=464).........AUC = 0.660
4-fold (N=464).........AUC = 0.621
5-fold (N=464).........AUC = 0.703
6-fold (N=465).........AUC = 0.742
7-fold (N=464).........AUC = 0.579
8-fold (N=464).........AUC = 0.641
9-fold (N=464).........AUC = 0.730
10-fold(N=464).........AUC = 0.704

Model:probit

Seed:1972

----------------------------------------------------------------
Cross-validated (cv) mean AUC, SD and Bootstrap Corrected 95%CI
----------------------------------------------------------------
cvMean AUC: | 0.6857
Booststrap corrected 95%CI: | 0.6348, 0.7079
cvSD AUC: | 0.0578
----------------------------------------------------------------

------------------------------------------------------------------
Mean cross-validated Sen, Spe and false(+) at lbw predicted values
------------------------------------------------------------------

Prevalence of lbw: 6.01%
------------------------

_Pp | _sen _spe _fp
-----+------------------------------
0.01 | 99.83 0.52 99.48
0.02 | 98.21 4.09 95.91
0.03 | 88.82 20.12 79.88
0.04 | 73.69 49.21 50.79
0.05 | 66.05 64.58 35.42
0.06 | 60.17 69.31 30.69

(output omitted ...)

0.39 | 0.27 99.98 0.02
------------------------------------------

*******************************************************
* Naive performance based on non-crossvalidated AUC *
*******************************************************

. logistic lbw mage medu mmarried prenatal fedu mbsmoke mrace order

Logistic regression Number of obs = 4,642
LR chi2(8) = 137.10
Prob > chi2 = 0.0000
Log likelihood = -986.35435 Pseudo R2 = 0.0650

------------------------------------------------------------------------------
lbw | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mage | .9959165 .0140441 -0.29 0.772 .9687674 1.023826
medu | .9451338 .0283732 -1.88 0.060 .8911276 1.002413
mmarried | .6109995 .1014788 -2.97 0.003 .4412328 .8460849
prenatal | .5886787 .073186 -4.26 0.000 .4613759 .7511069
fedu | 1.040936 .0214226 1.95 0.051 .9997838 1.083782
mbsmoke | 2.145619 .3055361 5.36 0.000 1.623086 2.836376
mrace | .3789501 .057913 -6.35 0.000 .2808648 .5112895
order | 1.05529 .0605811 0.94 0.349 .9429895 1.180964
_cons | .3468141 .1498299 -2.45 0.014 .1487176 .8087812
------------------------------------------------------------------------------

. predict fitted, pr

. roctab lbw fitted

ROC -Asymptotic Normal--
Obs Area Std. Err. [95% Conf. Interval]
------------------------------------------------------------
4,642 0.6939 0.0171 0.66041 0.72749

Authors

Miguel Angel Luque-Fernandez

LSHTM, NCDE, Cancer Survival Group, London, UK

Email: miguel-angel.luque@lshtm.ac.uk

Camille Maringe

LSHTM, NCDE, Cancer Survival Group, London, UK

Email: camille.maringe at lshtm.ac.uk

Daniel Redondo-Sanchez

Biomedical Research Institute of Granada (ibs.Granada)

Email: daniel.redondo.easp at juntadeandalucia.es

Acknowledgements

Miguel Angel Luque Fernandez is supported by the Spanish National Institute of Health, Carlos III Miguel Servet I Investigator
Award (CP17/00206).

References

Luque-Fernandez MA, Redondo-Sánchez D, Maringe C. cvauroc: Command to compute cross-validated area under the curve for ROC analysis after predictive modeling for binary outcomes. The Stata Journal. 2019;19(3):615-625. doi:10.1177/1536867X19874237

Miguel Angel Luque-Fernandez (2016), Crossvalidation in Epidemiology Presentation

StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP.

Hastie T., Tibshirani R., Friedman J., (2013). The elements of Statistical Learning, Data Mining, Inference and Prediction. Springer Series in Statistics.

Also see

Online: crossfold roctab lsens lroc rocreg