help kappaetc (SJ18-4: st0544)

Title

kappaetc -- Interrater agreement

Syntax

Interrater agreement, variables record raw ratings

kappaetc
varname1
varname2
[varname3 ...]
[if] [in]
[weight]
[, kappaetc_options]

Interrater agreement, variables record frequency of ratings

kappaetc
varname1
varname2
[varname3 ...]
[if] [in]
[weight], frequency [kappaetc_options]

Immediate command, interrater agreement, two raters, contingency table

kappaetci
#11 #12 [...] \
#21 #22 [...] [\ ...]
[, tab
kappaetc_options]

More

Choosing an appropriate method of analysis

Test differences of correlated agreement coefficients

Intraclass correlation coefficients (interrater and intrarater reliability)

Limits of agreement (Bland-Altman plot)

kappaetc_options

Main
wgt(wgtid [, wgt_options]) specify
how disagreements are weighted; see Options for
alternatives

se(se_type) specify how standard errors are estimated;
see Options for alternatives

frequency specify that variables record rating frequencies

categories(numlist) specify predetermined rating categories

listwise exclude subjects with missing ratings

Reporting
level(#) set confidence level; default is
level()

showweights display weighting matrix

benchmark[(benchmark_method [, benchmark_options])]
benchmark interrater agreement coefficients; see Options

showscale display benchmark scale

testvalue([relop]#) test whether coefficients are equal
to #; default is testvalue(0)

nociclipped do not clip confidence intervals at -1 and 1

noheader suppress output header

notable suppress coefficient table

format_options control column formats

Advanced
nsubjects(#) specify size of subject universe

nraters(#) specify size of rater population

largesample use standard normal distribution
for p-values and intervals

Immediate command
tab display contingency table

Miscellaneous
store(name|stub*) store
(additional) returned results under name or stub*

ttest perform paired t tests of differences between correlated
agreement coefficients; see kappaetc, ttest

icc(model) estimate intraclass correlation coefficients
(interrater
and intrarater reliability); see kappaetc, icc()

loa[(#)] estimate limits of agreement
and produce Bland-Altman plot; see kappaetc, loa

by is allowed only with kappaetc; see [D] by.

fweights and iweights are allowed only with kappaetc;
see weight.

Description

kappaetc estimates various interrater agreement coefficients, their
standard errors, and their confidence intervals. Statistics are estimated for
any number of raters, any number of rating categories, and in the presence of
missing values (that is, varying number of ratings per subject). Disagreement
among raters may be weighted by user-defined weights or a set of prerecorded
weights, suitable for any level of measurement.

The command is implemented using methods and formulas discussed in Gwet
(2014). It calculates percent agreement, the Brennan and Prediger (1981)
coefficient, Cohen's (1960, 1968) kappa and its generalization by Conger
(1980), Fleiss's (1971) kappa, Gwet's (2008, 2014) AC, and Krippendorff's
(1970, 2011, 2013) alpha coefficient.

Standard errors are estimated conditionally upon the sample of raters,
allowing results to be projected only to the subject universe. Optionally,
kappaetc calculates (jackknife) standard errors conditional on the
sample of subjects or unconditional standard errors, allowing projection of
results to both the subject universe and the rater population.

kappaetc assumes that each observation is a subject (unit of analysis)
and that variables contain the ratings by raters (coders, judges, observers).
Thus, the first variable records the ratings assigned by the first rater, the
second variable records the ratings assigned by the second rater, and so on.
With the frequency option, each observation is still assumed to
represent one subject. However, variables are expected to record the
frequencies of ratings.

kappaetc also assumes that all possible rating categories are observed
in the data. This assumption is crucial. If some of the rating categories
are not used by any of the raters, the full set of conceivable ratings must be
specified in the categories() option. Failing to do
so might produce incorrect results for all weighted agreement coefficients;
Brennan and Prediger's coefficient and Gwet's AC will be incorrectly estimated
even if no weights are used.

kappaetci calculates interrater agreement for two raters from a
contingency table of rating frequencies. The rows and columns of the table
are assumed to represent the rating categories, and the cell frequencies
indicate the number of subjects that have been classified into the respective
categories by the two raters. The syntax mirrors that of
tabi. Also see immed for a general description
of immediate commands.

Options

Main

wgt(wgtid [, wgt_options]) specifies that
wgtid be used to weight disagreements. The available weights and
wgt_options are described below.

identity weights are the q x q identity matrix, where q is the number
of categories used to rate subjects. Identity weights are the default and
result in the unweighted analysis.

ordinal weights are defined as 1 - comb(|k-l|+1),
2)/comb(q, 2) for all k!=l, where k and l represent the ranked categories 1,
2, ..., q and q is the number of rating categories. The wgt_option
krippendorff is allowed and specifies that ordinal weights suggested by
Krippendorff (2011) be used instead. The latter are defined as 1 - sum(n_g -
(n_k+n_l)/2)^2, where the n_* are the observed number of pairable values k and
l. Note that standard errors are not available with Krippendorff's ordinal
weights.

linear weights are defined as 1 - |k-l|/|q_max-q_min|, where k and l
refer to the actual ratings and q_max and q_min are the maximum and minimum of
all ratings. The wgt_option noabsolute is allowed and specifies
that k and l be interpreted as row and column indices of the weighting matrix.
w is a synonym for linear with wgt_option noabsolute.
These are the same weights as those used by kap.

quadratic weights are defined as 1 - (k-l)^2/(q_max-q_min)^2, where k
and l refer to the actual ratings and q_max and q_min are the maximum and
minimum of all observed ratings. The wgt_option noabsolute is
allowed and specifies that k and l be interpreted as row and column indices of
the weighting matrix. w2 is a synonym for quadratic with
wgt_option noabsolute. These are the same weights as those used by
kap.

radical weights are defined as 1 -
sqrt(|k-l|)/sqrt(|q_max-q_min|), where k and l refer to the actual
ratings and q_max and q_min are the maximum and minimum of all ratings. The
wgt_option noabsolute is allowed and specifies that k and l be
interpreted as row and column indices of the weighting matrix.

ratio weights are defined as 1 -
[(k-l)/(k+l)]^2/[(q_max-q_min)/(q_max+q_min)]^2, where k and l refer to the
actual ratings and q_max and q_min are the maximum and minimum of all ratings.
The wgt_option noabsolute is allowed and specifies that k and l be
interpreted as row and column indices of the weighting matrix.

circular [pi|180|#] weights are defined as 1 -
sin(angle*(k-l)/(q_max-q_min+1))^2/M, where angle is pi
radians if pi was specified or 180 degrees if 180 was specified
and where k and l refer to the actual ratings, q_max and q_min are the maximum
and minimum of all ratings, and M is the maximum of all weights; the sine
function's argument defaults to pi (that is,
c(pi)). When # is specified as 0 <= # < 1,
circular weights proposed by Warrens and Pratiwi (2016) are used instead. The
latter are defined as #*[(|k-l|==1)+(|k-l|==|q-1)] for all k!=l. When
# is specified, the wgt_option noabsolute is required and
specifies that k and l are interpreted as row and column indices of the
weighting matrix.

bipolar weights are defined as (k-l)^2/[M(k+l-2*q_min)*(2*q_max-k-l)],
where k and l refer to the actual ratings, q_max and q_min are the maximum and
minimum of all ratings, and M is the maximum of all weights.

power # weights are defined as 1 -
(|k-l|^#)/(|q_max-q_min|^#), where k and l refer to the actual
ratings and q_max and q_min are the maximum and minimum of all ratings. These
weights are discussed in Warrens (2014) as a generalization of identity
(#=0), linear (#=1), quadratic (#=2), and radical (#=0.5)
weights. The wgt_option noabsolute is allowed and specifies that
k and l be interpreted as row and column indices of the weighting matrix.

kapwgt are weights defined with the kapwgt command. The
wgt_option kapwgt is allowed and must be used if kapwgt has
the same name as one of the prerecorded weights (or their abbreviations)
discussed above.

matname are weights defined in a Stata matrix. The
wgt_option matrix is allowed and must be used if matname is
the same name as kapwgt or any of the prerecorded weights (or their
abbreviations) discussed above.

se(se_type) specifies how standard errors are estimated. Any
estimated interrater agreement coefficient potentially depends on two samples:
subjects to be rated might be drawn from a universe of subjects, while raters
might be drawn from a rater population. Standard errors may therefore be
conditional upon either of the samples, or unconditional, accounting for the
two respective sampling errors. The appropriateness of these different
standard errors depends on the research questions. Available se_types
are described below.

conditional [raters] are the default standard errors and are
estimated conditionally upon the sample of raters. These standard errors are
appropriate when results are to be generalized to the subject universe, given
the specific raters.

conditional subjects requests that standard errors be
estimated conditionally upon the sample of subjects. The extent of agreement
among all but one rater is obtained for each of the r raters in the sample.
Technically, these standard errors are implemented using a jackknife approach.
These standard errors allow projection of results to the rater population,
given the rated subjects.

unconditional standard errors are appropriate if the results are to
be projected to the universe of subjects and the rater population. They are
calculated as the square root of the sum of variances due to the subject and
rater sample.

frequency specifies that variables represent rating categories. The
first variable records the frequency of the first rating category, the second
variable records the frequency of the second rating category, and so on.
Rating categories are assumed to be the integer sequence 1, 2, ..., q (but see
the option categories()). Note that all possible
ratings must be represented by one variable even if the frequency is 0 for all
subjects. Cohen's (1960, 1968) and Conger's (1980) kappa cannot be calculated
from recorded rating frequencies, and only the default standard errors,
conditional on the rater sample, are available.

categories(numlist) specifies the predetermined rating
categories. By default, the set of rating categories is obtained from the
data. There are two situations where this option should be used.

When variables contain ratings (the default), the full set of possible rating
categories must be specified if not all of them are observed in the data.
Failing to do so may lead to incorrect results. The order in which rating
categories are specified does not matter; categories are sorted internally.
Note that noninteger values are processed in double
precision. To convert them to float precision, specify
categories(float(numlist)).

With the frequency option, the ratings are assumed to be the integer
sequence 1, 2, ..., q that corresponds to the specified variables. Likewise,
with the immediate form of the command, the ratings are assumed to be the
integer sequence 1, 2, ..., q of rows and columns entered. In both cases, the
categories() option may be used to specify alternative rating
categories, including noninteger, negative, and even missing values. Also in
both cases, the order in which the rating categories are specified matters and
corresponds to the respective variables or sorted values underlying the table.

listwise specifies that subjects with missing ratings be excluded from
the analysis. By default, all subjects that are rated by at least one (two,
for Krippendorff's alpha) rater or raters are used to estimate expected
agreement. Observed agreement is based only on those subjects that are rated
by two or more raters. casewise is a synonym for listwise.

Reporting

level(#) specifies the confidence level, as a percentage, for confidence
and benchmark intervals. The default is level().

showweights additionally displays the weighting matrix below the
coefficient table. For the unweighted analysis, the identity matrix is shown.

benchmark[(benchmark_method [,
benchmark_options])] benchmarks the estimated interrater agreement
coefficients using the Landis and Koch (1977) scale and the method proposed by
Gwet (2014).

Landis and Koch (1977) suggest the following benchmark scale for interpreting
the kappa statistic:

<0.00 Poor

0.00-0.20 Slight

0.21-0.40 Fair

0.41-0.60 Moderate

0.61-0.80 Substantial

0.81-1.00 Almost Perfect

Gwet (2014) argues that the probability distribution of an agreement
coefficient depends on the number of subjects, raters, and categories in the
study. Therefore, the error margin associated with these sources of variance
must be accounted for when comparing estimated coefficients with predetermined
thresholds. Consequently, benchmarking should be probabilistic rather than
deterministic. He proposes a statistical method that consists of three steps.
First, the probability for an agreement coefficient to fall into each of the
intervals, defined by the benchmark limits, is calculated. Next, the
cumulative probability for the intervals, starting from the highest level, is
computed. Finally, the interval associated with a cumulative probability
larger than a given threshold (% by default) determines the
benchmark level.

With the benchmark option, kappaetc displays the estimated
coefficients and their standard errors. It reports the probability for each
coefficient to fall into the selected benchmark interval along with the
cumulative probability exceeding the predetermined threshold associated with
this interval. The interval limits are shown as well.

Available benchmark_methods are described below.

probabilistic is the default method and selects the benchmark interval
associated with the smallest cumulative membership probability exceeding
c(level). The threshold is controlled by the level() option.

deterministic selects the benchmark interval associated with the
estimated agreement coefficient. This method is deterministic in that the
chosen interval is determined solely by the point estimate, ignoring any
uncertainty associated with its estimation.

With both benchmark_methods, the following benchmark_option is
allowed.

scale(spec) specifies the benchmark scale. spec is usually one of
landis (or koch), fleiss, or altman. The default is
scale(landis) (or scale(koch)) and results in the Landis and Koch
scale as shown above. fleiss requests a three-level scale, suggested by
Fleiss, Levin, and Paik (2003), and altman collapses the first two
levels of the default scale into one category yielding the Altman (1991)
scale. Alternatively, spec explicitly specifies the (upper-limit)
benchmarks as a numlist. The Landis and Koch scale could be
obtained as scale(0(.2)1).

showscale additionally displays the benchmark scale for interpreting
coefficients. This option is ignored when benchmark is not specified.

testvalue([relop]#) tests whether the
estimated agreement coefficients equal #. The default is
testvalue(0). relop is one of the relational operators
>[=] or <[=] and performs one-sided tests.

nociclipped reports confidence intervals as estimated. The default is
to restrict confidence limits to fall into the range of -1 < # < 1.

noheader suppresses the report about the number of subjects, ratings per
subject, and rating categories. Only the coefficient table is displayed.

notable suppresses the display of the coefficient table.

format_options are any of the following:

cformat(%fmt) specifies how to format
coefficients, standard errors, and confidence limits. The maximum format
width is 8.

pformat(%fmt) specifies how to format
p-values. The maximum format width is 5.

sformat(%fmt) specifies how to format test
statistics. The maximum format width is 6.

Advanced

nsubjects(#) specifies the size of the subject universe to be used for
the finite sample correction. The default is nsubjects(.), leading to a
sampling fraction of 0 that is assumed to be negligible. This option is
seldom used.

nraters(#) specifies the size of the rater population to be used for the
finite sample correction. The default is nraters(.), leading to a
sampling fraction of 0 that is assumed to be negligible. This option is
relevant only for standard errors that are conditional on the sample of
subjects or unconditional standard errors. It is seldom used although the
default might overestimate the variance for small rater populations.

largesample specifies that the calculation of p-values and intervals be
based on the standard normal distribution rather than the t distribution.
This is the default for unconditional standard errors. largesample is a
reporting option and it is seldom used.

Immediate command

tab displays the two-way table of cell frequencies. The option is
useful for data entry verification.

Miscellaneous

store(name|stub*) returns additional results in
r() and stores them under name. This option is intended for use
with the ttest option (see below). Results are stored using
_return hold. Note that any results previously held under
name will be overwritten. When the option store() is combined with
the by prefix, results are stored under stub1 ...
stubq.

ttest performs paired t tests of correlated agreement coefficients. See
kappaetc, ttest.

icc(model) estimates intraclass correlation coefficients as a measure of
interrater and intrarater reliability. See
kappaetc, icc().

loa[(#)] estimates # percent limits of agreement
and produces a Bland-Altman plot. See kappaetc, loa().

Remarks

Remarks are presented under the following headings:

Relation to official Stata's kap and kappa commands

Equivalent agreement coefficients

Relation to official Stata's kap and kappa commands

The percent agreement that is reported by kappaetc is the same as the
observed agreement reported by Stata's kap command.
kappaetc additionally estimates a standard error and confidence interval
for the observed agreement.

For two unique raters and no missing ratings, both kappaetc and Stata's
kap command estimate the same Cohen's kappa coefficient. However, the
standard error and p-value differ between the two commands. Stata's kap
command implements a model-based (analytic) approximate formula for the
standard error and reports a one-sided test based on the standard normal
distribution. kappaetc implements a design-based formula to obtain the
standard error and reports a two-sided test based on the t distribution with
n-1 degrees of freedom. kappaetc additionally provides a confidence
interval.

For more than two (nonunique) raters and no missing ratings (that is, a
constant number of ratings per subject), both kappaetc (possibly with
the frequency option) and Stata's kap (or kappa) command
estimate the same Fleiss kappa coefficient. However, kappaetc does not
report a kappa coefficient for each rating category. Standard errors and
p-values differ between the commands for the same reasons explained for the
two unique raters case above.

Equivalent agreement coefficients

The Brennan and Prediger coefficient, estimated by kappaetc, is
essentially equivalent to Bennet, Alpert, and Goldstein's (1954) S, the G
Index (Holley and Guilford 1964), the random-error coefficient (Maxwell 1970,
1977), Janson and Vegelius's (1979) C, Perreault and Leigh's (1989) Ir, and
Byrt, Bishop, and Carlin's (1993) prevalence-adjusted and bias-adjusted kappa,
also known as PABAK (compare Gwet [2014, 30, 65-69]).

Fleiss's kappa reduces to Scott's pi (1955) in the case of two raters (compare
Gwet [2014, 53]). The latter is identical to Byrt, Bishop, and Carlin's
(1993) bias-adjusted kappa, also known as BAK (compare Gwet [2014, 69]).

Examples

Examples are drawn from those in kap and
[R] kappa.

Two raters

. webuse rate2

. kappaetc rada radb

. kappaetc rada radb, wgt(linear)

. kapwgt xm 1 \ .8 1 \ 0 0 1 \ 0 0 .8 1

. kappaetc rada radb, wgt(xm, kapwgt)

More than two raters, varying number of ratings per subject (missing
values)

. webuse rvary2

. kappaetc rater1-rater5

Stored results

kappaetc stores the following in r():

Scalars

r(N) number of subjects

r(r) number of raters (maximum number of ratings per subject)

r(r_min) minimum number of ratings per subject

r(r_avg) average number of ratings per subject

r(r_max) maximum number of ratings per subject
(same as r(r))

r(jk_miss) number of missing jackknife coefficients

r(level) confidence level

Macros

r(cmd) kappaetc

r(wtype) weight type

r(wexp) weight expression

r(wgt) wgtid for weighting disagreement

r(userwgt) kapwgt or matrix
(only with user-defined wgtid)

r(setype) conditional or unconditional

r(seconditional) raters or subjects

Matrices

r(b) coefficient vector

r(table) information from the coefficient table

r(se) standard errors

r(b_jknife) jackknife coefficients

r(se_cond_subjects) standard errors
conditional on subjects

r(se_cond_raters) standard errors
conditional on raters

r(df) coefficient-specific degrees of freedom

r(prop_o) observed proportion of agreement

r(prop_e) expected proportion of agreement

r(W) weighting matrix for disagreement
(note capitalization)

r(categories) distinct levels of ratings

r(estimable) whether coefficient could be estimated

r(table_benchmark_prob) information from the probabilistic
benchmark table

r(table_benchmark_det) information from the deterministic
benchmark table

r(benchmarks) upper limits of benchmark intervals

r(imp) probability to fall into an interval

r(p_cum) cumulative interval membership probability

r(weight_i) subject-level weights

r(b_istar) subject-level agreement coefficients

Acknowledgments

I am deeply grateful to Kilem Gwet for continuous support and patiently
clarifying my questions during the implementation of kappaetc.

The name kappaetc is borrowed from entropyetc with approval from
Nicholas Cox (2016).

References

Altman, D. G. 1991. Practical Statistics for Medical Research. London:
Chapman & Hall.

Bennet, E. M., R. Alpert, and A. C. Goldstein. 1954. Communications through
limited response questioning. Public Opinion Quarterly 18: 303-308.

Brennan, R. L., and D. J. Prediger. 1981. Coefficient kappa: Some uses,
misuses, and alternatives. Educational and Psychological Measurement 41:
687-699.

Byrt, T., J. Bishop, and J. B. Carlin. 1993. Bias, prevalence and
kappa. Journal of Clinical Epidemiology 46: 423-429.

Cohen, J. 1960. A coefficient of agreement for nominal
scales. Educational and Psychological Measurement 20: 37-46.

------. 1968. Weighted kappa: Nominal scale agreement with provision for
scaled disagreement or partial credit. Psychological Bulletin 70:
213-220.

Conger, A. J. 1980. Integration and generalization of kappas for multiple
raters. Psychological Bulletin 88: 322-328.

Cox, N. J. 2016. entropyetc: Stata module for entropy and related measures for categories. Statistical Software Components S458272, Department of Economics, Boston College. https://ideas.repec.org/c/boc/bocode/s458272.html.

Fleiss, J. L. 1971. Measuring nominal scale agreement among many
raters. Psychological Bulletin 76: 378-382.

Fleiss, J. L., B. Levin, and M. C. Paik. 2003. Statistical Methods for Rates and Proportions. 3rd ed. Hoboken, NJ: Wiley.

Gwet, K. L. 2008. Computing inter-rater reliability and its variance in the
presence of high agreement. British Journal of Mathematical and Statistical Psychology 61: 29-48.

------. 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD:
Advanced Analytics.

Holley, J. W., and J. P. Guilford. 1964. A note on the G index of agreement. Educational and Psychological Measurement 24: 749-753.

Janson, S., and J. Vegelius. 1979. On generalizations of the G index and the PHI
coefficient to nominal scales. Multivariate Behavioral Research 14:
255-269.

Krippendorff, K. 1970. Estimating the reliability, systematic error and
random error of interval data. Educational and Psychological Measurement
30: 61-70.

------. 2011. Computing Krippendorff's
alpha-reliability. https://repository.upenn.edu/asc_papers/43/.

------. 2013. Content Analysis: An Introduction to Its Methodology. 3rd
ed. Thousand Oaks, CA: Sage.

Landis, J. R., and G. G. Koch. 1977. The measurement of observer agreement
for categorical data. Biometrics 33: 159-174.

Maxwell, A. E. 1970 Comparing the classification of subjects by two
independent judges. British Journal of Psychiatry 116: 651-655.

------. 1977. Coefficient of agreement between observers and their
interpretation. British Journal of Psychiatry 130: 79-83.

Perreault, W. D., and L. E. Leigh. 1989. Reliability of nominal data based on
qualitative judgments. Journal of Marketing Research 26: 135-148.

Scott, W. A. 1955. Reliability of content analysis: The case of nominal
scale coding. Public Opinion Quarterly 19: 321-325.

Warrens, M. J. 2014. Power weighted versions of Bennett, Alpert, and
Goldstein's S. Journal of Mathematics 2014: 231909.

Warrens, M. J., and B. C. Pratiwi. 2016. Kappa coefficients for circular
classifications. Journal of Classification 33: 507-522.

Author

Daniel Klein

International Centre for Higher Education Research Kassel

Kassel, Germany

klein@incher.uni-kassel.de

Also see

Article: Stata Journal, volume 18, number 4: st0544

Help: [R] kappa, [R] icc, kappa2, kapci,
kappci, kanom, kalpha, krippalpha, kapssi,
concord, classtabi, entropyetc (if installed)