*! version 0.2 2024-02-16

Title

cpt -- Optimal cut-points for empirical ROC curves and other ROC/AUC calculations

Syntax

cpt
varlist(min=2)
[,
options]

Optional
format(string) Stata format for the cut-point values in the returned
ROC matrix

replace Option for replacing generated variables with sensitivity
and specificity

bamber calculate standard errors by using the Hanley method. See roctab

hanley calculate standard errors by using the Bamber method. See roctab

binomial calculate exact binomial confidence intervals. See roctab

cv(#) do cross-validation in # subgroups by predicting ROC values
in each group by the groups.
CV must be an integer greater than or equal to 2 or zero.
Default value is 0, i.e., no cross-validation.

reps(#) average # repetions of the predictions for the ROC.
Default value is 1, i.e., no averaging.

seed Set seed value for the options cv and reps.

graph Generate a default graph.

twoway options Generate a default graph with the twoway options.
Option graph is not necessary in this case.

Description

The command cpt is a wrapper for roctab.

It generates three matrices: A AUC estimation report;
The sensitivity, the specificity, the accuracy, the lr+, and the lr- from
roctab at each cut-point value, as well as the PPV, the NPV, and
the AUC in each cut-point value.

AUC is the average of sensitivity and specificity and is the AUC in the
cut-point values.

PPV is the prevalence weighted average of the sensitivity and the 1 - -specificity.

NPV is the prevalence weighted average of the 1 - sensitivity and the specificity.

The cpt command marks the optimal cut-point values by Youden (J) and
Liu(L) in the returned ROC matrix row names and the submatrix of the ROC matrix
containing only the optimal cut-point values.

Finally, variables (prefix tpr_) for sensitivity and false positive rates
(prefix fpr_) are saved to make one or more curves.

ROC curves, AUC, Cross-validation, and repetions

An ROC curve is a graphical representation that illustrates the
performance of a binary classification model across different thresholds.
It plots the True Positive Rate (TPR) on the Y-axis against the False Positive
Rate (FPR) on the X-axis. The top-left corner of the ROC plot represents the
"ideal" point (TPR = 1, FPR = 0), which is not very realistic but indicates
better performance.
A larger Area Under the Curve (AUC) usually indicates better model performance.
The "steepness" of the ROC curve matters: We aim to maximize TPR while
minimizing FPR. Methods like Youden and Liu are ways of finding these
optimal pairs of FPR and TPR.

Cross-validation is a technique used to assess the performance of a model
by dividing the dataset into multiple subsets (folds), typically of equal sizes.
We use each fold in shifts as a validation set while we use the remaining folds
for training.
Repeating this process with different subsets, we get a more robust estimate
of model performance.
Cross-validation helps estimate the variance of model performance metrics
(such as AUC) due to variations in training data.
It provides insights into how well the model generalizes to unseen data.
Cross-validation helps prevent overfitting by assessing model performance on
different data subsets.

Using a single cross-validation once is affected by random variations in
data splitting.
We reduce bias and obtain more precise estimates by repeating cross-validation
multiple times (e.g., 100 repeats of 10-fold cross-validation).
In summary, ROC curves, cross-validation, and repetition are essential tools
for evaluating and understanding the performance of classification models.

Examples

Getting example data:

`"webuse hanley"'

Calling cpt:

`"cpt disease rating"'

Calling cpt with roctab options:

`"cpt disease rating, binomial format(%2.0f) replace"'

The returned ROC matrix:

`"matlist r(roc)"'

The returned AUC matrix:

`"matlist r(auc)"'

The returned matrix with optimal cut-point values:

`"matlist r(cutpt)"'

A simple ROC curve:

`"line tpr_p_disease fpr_p_disease, name(crude, replace)"'

Calling cpt with cross-validation in 10 random groups:

`"cpt disease rating, replace cv(10)"'

Calling cpt using the average of 20 cross-validations in 10 random groups:

`"cpt disease rating, replace cv(10) reps(20)"'

The ROC curve based on cross-validation and reptions:

`"line tpr_p_disease fpr_p_disease, name(cv10reps20, replace)"'

Stored results

Scalars:

r(Youden_p) Youden's optimal p-value.

r(Youden_v) If there is one explanatory variable.
Youden's optimal p-value converted back to the variable.

r(Liu_p) Liu's optimal p-value.

r(Liu_v) If there is one explanatory variable.
Liu's optimal p-value converted back to the variable.

Macros:

r(auctext) AUC report from roctab

r(graph_cmd) The twoway graph command generating the graph.

r(Youden_text) If there is one explanatory variable.
Youden's optimal p-value converted back to the variable with orientation as text.

r(Liu_v) If there is one explanatory variable.
Liu's optimal p-value converted back to the variable with orientation as text.

Matrices:

r(auc) AUC report from roctab

r(roc) Sensitivity, specificity, PPV, NPV, accuracy, lr+, and lr- from
roctab at each cut-point values.
Also the AUCs in each cut-point value are reported.
The optimal cut-point values by Youden (J) and Liu(L) are marked in the matrix row
names

r(cutpt) A submatrix of the ROC matrix containing only the
optimal cut-point values

Authors and support

Author:

Niels Henrik Bruun,

Aalborg University Hospital

Support:

niels.henrik.bruun@gmail.com