help rioc

Title

rioc -- Relative improvement over chance (RIOC)

Syntax

Relative improvement over chance

rioc
prediction
outcome
[if] [in]
[weight]
[, options ]

Immediate command

rioci
#TP #FP [\] #FN #TN
[, notab
options ]

where prediction (also: classvar) holds the predicted state
(classification) and outcome (also: refvar) holds the true
(actual) state.

#TP is the number of true positives, #FP is the
number of false positives, #FN is the number of false
negatives, and #TN is the number of true negatives.

tab display contingency table

detail calculate the total (observed), chance (expected),
and maximum possible proportion of correctly classified subjects

stats(stat) calculate additional proportions

kappa calculate Cohen's kappa

sezero calculate the standard error for testing against zero

smallsample calculate confidence interval for RIOC in
small samples

level(#) set confidence level; default is
level()

format_options control column formats

by is allowed only with rioc; see [D] by.

fweights are allowed only with rioc;
see weight.

Description

rioc calculates the relative improvement over chance (RIOC) coefficient
for 2 x 2 tables (Farrington & Loeber 1989, Copas & Loeber 1990). The RIOC
corrects the observed proportion of correctly classified subjects for chance
and maximum ceiling.

The command is implemented using methods and formulas discussed in
Copas and Loeber (1990) and in Farrington and Loeber (1989). In both
variables prediction and outcome, values equal to nonzero
and nonmissing (typically values of one) indicate a true (positive)
state; values equal to zero indicate a false (negative) state.

Remarks

Although the syntax diagram implies that the order of the variables
is always relevant, the relative improvement over chance (RIOC) does
not depend on the orientation of the contingency table. The orientation
of the contingency table does affect other statistics, such as the
sensitivity and specificity.

Options

tab displays the contingency table. This is the default for the
immediate command, rioci; specify notab to suppress the
contingency table.

detail calculates the total (observed), chance (expected), and
maximum possible proportion of correctly classified subjects. Confidence
intervals are calculated using Stata's cii command with the
default exact method.

stats(stat) requests additional proportions. Confidence intervals
are calculated using Stata's cii with the default exact method;
stat is one or more of

TPR true positive rate (sensitivity)

TNR true negative rate (specificity)

PPV positive predictive value

NPV negative predictive value

all all of the above

Note that all predefined stats require that the variables are
specified in correct order: prediction outcome.

stats(name: numerator/denominator) is
an alternative syntax for stats() that defines proportions to be
calculated. Confidence intervals are calculated using Stata's cii
with the default exact method. name is a valid name for the proportion,
and numerator and denominator are expressions
that typically refer to the cells of the contingency table. For example, in
numerator and denominator, a refers to the number of true
positives and n refers to the total number of subjects. The notation
follows Copas and Loeber (1990):

a true positives (synonym: TP)

b false positives (synonym: FP)

c false negatives (synonym: FN)

d true negatives (synonym: TN)

e a + b (synonym: r1)

f a + c (synonym: c1)

n a + c + b + d (synonym: N)

More than one stat may be defined. For example, sensitivity (the true
positive rate) and specificity (the true negative rate) may be obtained as

. rioc ... , stats(Sens: TP/(TP+FN) Spec: TN/(TN+FP))

Note that both numerator and denominator must be enclosed in
parentheses if they contain spaces. More technically, the defined stats
consist out of 5 tokens: name, :, numerator, /, and
denominator.

Although, technically, rioc can compute risk-ratios, e.g.,
stats(LRpos: (a*(n-f))/(f*b)), the confidence intervals
are still calculated using cii; for risk-ratios, confidence
intervals are not valid (see csi).

kappa calculates Cohen's kappa. The standard error and the confidence
interval are implemented according to Fleiss et al. (1969). Option
sezero may be specified to obtain the same standard error as Stata's
kap command.

sezero calculates the standard error of RIOC (and kappa, if
kappa is also specified) for testing against zero. For the RIOC,
the p-value matches the p-value of Pearson's chi-squared test with 1
degree of freedom; for kappa, the standard error and z-value match those
of Stata's kap commmand. Note that confidence intervals are not
affected by sezero.

smallsample calculates the confidence interval for RIOC in small
samples. smallsample only affects the confidence interval for the
RIOC; no confidence intervals are calculated for additional
proportions and statistics. Also, when smallsample is specified,
standard errors are not calculated, unless sezero is also specified.

level(#) specifies the confidence level, as a percentage, for
confidence intervals. The default is level()
(see c(level)).

cformat(%fmt) specifies how to format
coefficients, standard errors, and confidence limits. The maximum format
width is 9.

pformat(%fmt) specifies how to format
p-values. The maximum format width is 5.

sformat(%fmt) specifies how to format
test statistics. The maximum format width is 8.

Examples

Example 1: Table I (Farrington & Loeber 1989:202)

. rioci 75 125 \ 25 175

Example 1a: Keep the dataset in memory (Caution: clears the dataset)

. rioci 75 125 \ 25 175 , replace

Example 2: More details, suppress table

. rioc prediction outcome [fweight=pop] , notab detail

Example 3: Add Sensitivity and Specificity

. rioc prediction outcome [fweight=pop] , tab detail stats(TPR TNR)

Example 4: Add prevalence

. rioc prediction outcome [fweight=pop] , stats(Prevalence: (TP+FN)/N)

Example 5: Replicate results from estat classification
(Caution: clears data)

. sysuse nlsw88 , clear

. logit union i.collgrad

. estat classification , cutoff(`=1/(1+exp(-_b[_cons]))')

. rioc collgrad union , stats(all FPR_0: b/(n-f) FNR_1: c/f FPR_1: b/e FNR_0: c/(n-e)) percent

Saved results

rioc saves the following in r():

Scalars

r(N) number of subjects

r(level) confidence level

r(rioc) relative improvement over chance (RIOC)

Macros

r(cmd) rioc

Matrices

r(table) information from the coefficients table

References

Cairney, J., & Streiner, D. L. 2011. Using relative improvement over
chance (RIOC) to examine agreement between tests: Three case examples
using studies of developmental coordination disorder (DCD) in
children. Research in Developmental Disabilities, 32, 87--92.

Copas, J. B., & Loeber, R. 1990. Relative improvement over chance
(RIOC) for 2 x 2 tables. British Journal of Mathematical and
Statistical Psychology, 43, 293--307.

Farrington, D.P., & Loeber, R. 1989. Relative improvement over chance
(RIOC) and phi as measures of predictive efficiency and strength of
association in 2 x 2 tables. Journal of Quantitative Criminology,
5, 201--213.

Fleiss, J. L., Cohen, J., & Everitt, B. S. 1969. Large sample standard
errors of kappa and weighted kappa. Psychological Bulletin, 72, 323--327.

Acknowledgments

A first version of rioc appeared on
Statalist as
part of a discussion with Rich Goldstein, Eric Melse, and Mike Lacy.

Author

Daniel Klein

University of Kassel

klein.daniel.81@gmail.com

Also see

Online: tabulate, ci, estat classification,
kap

if installed: diagt, classtabi, kappaetc, riocplot