help kappaetcssi (SJ18-4: st0544)

Title

kappaetcssi -- Approximate sample-size estimation
for agreement coefficients

Syntax

kappaetcssi
E
[prop_o]
[, options]

where E is the desired error margin, that is, half the width of the
confidence interval, and prop_o is the target observed proportion
of agreement.

level(#) set confidence level; default is level()

cformat(%fmt) control display format

noreturn do not return results

nsubjects(#) specify size of subject universe

largesample use standard normal distribution for standard error
and error margin

Description

kappaetcssi estimates an approximate required sample size for agreement
coefficients so that the error margin (half the width of the confidence
interval) remains below a specified value. See immed for a general
description of immediate commands.

The command implements a simple convenient procedure outlined in Gwet (2014,
158-160). Instead of deriving exact formulas for various specific agreement
coefficients, this procedure focuses on the observed proportion of agreement,
denoted prop_o. For any two raters, the large-sample normal
approximation of its variance is V = prop_o(1-prop_o)/n,
and the associated error margin is E = z*sqrt(V), where
z is the negative of the ((1-level(#)/100)/2) quantile of the
normal distribution. The optimal number of subjects is then n =
(z^2*V)/E^2.

kappaetcssi takes as arguments the desired error margin and, optionally,
the expected observed proportion of agreement. The latter defaults to 0.5 if
not specified. The command estimates the sample size that results in an error
margin less than or equal to the specified value. It also estimates the large
sample standard error and error margin associated with the observed proportion
of agreement for sample size n. Additionally, the standard error and
error margin for small samples based on the t distribution with n-1
degrees of freedom is reported.

Options

level(#) specifies the confidence level, as a percentage, for confidence
intervals. The default is level().

cformat(%fmt) specifies how to format
coefficients, standard errors, and error margins. The maximum format width is
8.

noreturn does not return results in r().

nsubjects(#) specifies the size of the subject universe to be used for
the finite sample correction. The default is nsubjects(.), leading to a
sampling fraction of 0 that is assumed to be negligible. This option affects
only the estimated standard error and error margin for small samples and is
seldom used.

largesample specifies that the calculation of the error margin be based
on the standard normal distribution rather than the t distribution. This
option affects only the error margin for small samples and is seldom used.

Remarks

The sample size that kappaetcssi reports is comparable with the results
obtained from power oneproportion when power is set to 0.5. For
example,

. kappaetcssi 0.1

gives the same result as

. power oneproportion 0.5 0.6 , power(0.5)

Examples

Estimate approximate required sample size for a maximum error margin of
0.2.

. kappaetcssi 0.2

Do the same as above but set expected observed proportion of agreement to
0.8.

. kappaetcssi 0.2 0.8

Stored results

kappaetcssi stores the following in r():

Scalars

r(level) confidence level

r(E) desired error margin

r(prop_o) observed proportion of agreement

r(N_pop) population size (only with nsubjects())

r(N) number of subjects

r(se) standard error (normal approximation)

r(errmarg) error margin (normal approximation)

r(se_t) standard error (small sample)

r(errmarg_t) error margin (small sample)

Reference

Gwet, K. L. 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed.
Gaithersburg, MD: Advanced Analytics.

Author

Daniel Klein

International Centre for Higher Education Research Kassel

Kassel, Germany

klein@incher.uni-kassel.de

Also see

Article: Stata Journal, volume 18, number 4: st0544

Help: [R] kappa, [PSS] power, kapssi, sskapp (if installed)