*! version 1.0.0 20Sept2023

Title

power kappa -- Power analysis for the two-rater kappa statistic with two or more categories

Syntax

Compute sample size

power kappa kappa0 kappa1
, marg(numlist) [ power(numlist) alpha(numlist) onesided graph[(graphopts)] ]

Compute power

power kappa kappa0 kappa1
, marg(numlist) [ n(numlist) alpha(numlist) onesided graph[(graphopts)] ]

where kappa0 is the null (hypothesized) kappa and
kappa1 is the alternative (target) kappa. kappa0 and kappa1 may each be
specified either as one number or as a list of values in parentheses
(see numlist).

marg(numlist) marginal probabilities (frequencies) of assignment into each category. The category frequencies must sum to 1.0; marg() is required

* alpha(numlist) significance level; default is alpha(0.05)

* power(numlist) power; default is power(0.80)

* n(numlist) total sample size; required to compute power

onesided one-sided test; default is two sided

graph[(graphopts)] graph results; see [PSS-2] power, graph

* Specifying a list of values in at least two starred options, or
two command arguments, or at least one starred option and one argument
results in computations for all possible combinations of the values; see
numlist.

Description

power kappa computes power and sample size for the kappa test of agreement between two raters when there are two or more categories
from which to choose from. power kappa computes power and sample size using the method described by Flack et al. (1988).

Options

marg(numlist) required. The marginal probabilities of rating a subject (target) into k categories. The category frequencies must sum to 1.0. As an example,
Cohen (1968) uses psychiatric diagnosis agreement to illustrate measures of interrater agreement. There are three diagnostic categories: personality
disorder (PD), neurosis (N), and psychosis (PS). The data have the two judges' estimated frequencies of rating subjects in each of the three categories
as (0.50, 0.30, 0.20) and (0.60, 0.30, 0.10). While these marginal ratings frequencies are not identical to one another, they are similar enough to
justify the assumption that the true marginals are roughly the same for the two judges.

alpha(numlist) sets the significance level of the test. The
default is alpha(0.05).

power(numlist) specifies the desired power at which sample size is to be computed.
If power() is specified in conjunction with n(),
then the actual power of the test is presented.

n(numlist) specifies the total number of subjects in the study to be used for determining power.

onesided indicates a one-sided test. The default is two sided.

graph, graph(); see [PSS-2] power, graph.

Examples

Examples: Computing sample size

Using the first row of data from Table 2 in Flack et al. (1988), we will compute the sample size required to detect a kappa of 0.60 given a
kappa of 0.40 under the null hypothesis for 3 categories with marginal probabilities of 0.50 0.26 0.24. We specify a one-sided test,
a 5% significance level and 80% power (the default).

. power kappa 0.40 0.60, marg(0.50 0.26 0.24) alpha(0.05) onesided

Same as above, using a power of 90% and a two-sided test

. power kappa 0.40 0.60, marg(0.50 0.26 0.24) alpha(0.05) power(0.90)

Same as above, but testing power at both 80% and 90% and alpha at the 0.01 and 0.05 level

. power kappa 0.40 0.60, marg(0.50 0.26 0.24) alpha(0.01 0.05) power(0.80 0.90)

Same as above, but applying a range of kappa values under the alternative hypothesis
and setting alpha levels to 0.05 and 0.01; and graphing the results

. power kappa 0.40 (0.60(0.05).95), marg(0.50 0.26 0.24) power(0.80) alpha(0.01 0.05) graph

Examples: Computing power

For a total sample of 93 subjects, compute the power of a one-sided test to
detect a kappa of 0.60 given a null kappa of 0.40 for 3 categories with marginal
probabilities of 0.50 0.26 0.24, at a 5% significance level (the default)

. power kappa 0.40 0.60, marg(0.50 0.26 0.24) n(93) onesided

Same as above but test alpha levels of 0.01, 0.05 and 0.10

. power kappa 0.40 0.60, marg(0.50 0.26 0.24) n(93) onesided alpha(0.01 0.05 0.10)

Compute powers for a range of sample sizes at alpha 0.01 and 0.05,
graphing the results

. power kappa 0.40 0.60, marg(0.50 0.26 0.24) n(70(5)160) onesided alpha(0.01 0.05) graph

Stored results

power kappa stores the following in r():

Scalars

r(alpha) significance level

r(kappa0) null kappa

r(kappa1) alternative kappa

r(tau0) max std error for the null kappa

r(tau1) max std error for the alternative kappa

r(beta) probability of a type II error

r(delta) effect size

r(divider) 1 if divider is requested in the table, 0 otherwise

r(N) total sample size

r(onesided) 1 for a one-sided test, 0 otherwise

r(power) power

Macros

r(type) test

r(method) kappa

r(columns) displayed table columns

r(labels) table column labels

r(widths) table column widths

r(formats) table column formats

Matrices

r(pss_table) table of results

References

Cohen, J. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial
credit. Psychological Bulletin 70: 213-220.

Flack, V. F., Afifi, A. A., Lachenbruch, P. A., and H. J. A. Schouten. 1988. Sample size determinations for the two
rater kappa statistic. Psychometrika 53: 321-325.

Citation of power kappa

power kappa is not an official Stata command. It is a free contribution
to the research community, like a paper. Please cite it as such:

Linden A. (2023). POWER KAPPA: Stata module to compute power and sample size for the two-rater kappa statistic with two or more categories

Authors

Ariel Linden

President, Linden Consulting Group, LLC

alinden@lindenconsulting.org

Also see

Online: kappa, power, kapssi (if installed), sskdlg (if installed)