help kalpha

kalpha has been superseded by kappaetc that is available from
the SSC archives. kappaetc estimates Krippendorff's alpha
and other inter-rater agreement coefficients along with their standard errors
and confidence intervals. kalpha continues to work but there will not
be any further updates for the software.

Title

kalpha -- Krippendorff's Alpha-Reliability

Syntax

Calculate Krippendorff's alpha coefficient

kalpha varlist [if] [in] [ , options ]

Create a variable containing Krippendorff's alpha

egen [type] newvar = kalpha(varlist)
[if] [in] [, options]

scale(metric) specify data metric (level of
measurement). Default is interval for numeric variables and
nominal for string variables

transpose specify that variables are observers (raters,
coders, judges, ...) and observations are units of analysis (subjects,
objects, ...)

bootstrap[(boot_options)] bootstrap
confidence interval and probability to fail to reach a minimum alpha

by is allowed; see [D] by.

Description

kalpha calculates Krippendorff's alpha reliability coefficient
(Hayes and Krippendorff 2007; Krippendorff 2004, 2011, 2013a).

kalpha assumes that each observation is an observer (rater, coder,
judge, ...) and that variables are units of analysis (subjects, objects,
...). Thus, the first observation identifies the first observer, the
second observation identifies the second rater, and so on. The first
variable is the first unit of analysis, the second variable is the second
unit of analysis, and so on.

In most datasets, observations are units of analysis (subjects, objects, ...)
and variables are observers (raters, coders, judges, ...). In this case, the
transpose option must be specified.

kalpha assumes that numeric variables are measured on interval scale
and string variables are measured on nominal scale. See option
scale() below.

The bootstrap option implements the algorithm described in Krippendorff
(2013b). This algorithm has recently been criticized by Gwet (2015) and Zapf
et al. (2016).

Options

scale(metric) specifies the data's metric or level of
measurement. Default metric is interval for numeric variables and
nominal for string variables. kalpha also supports ordinal,
ratio, circular and polar data.

For circular data metric may be specified as
circular[(U)], where U specifies the
fixed circumference (i.e. number of equal intervals) of the circle.
If U is not specified it defaults to
max - min + 1, where max and min are
the observed maximum and minimum values in the data. Differences are
calculated in radiance. To calculate differences in degrees, specify
circulardeg[(U)].

For polar data metric may be specified as
polar[(min max)], where min and max
specify the minimum and maximum value of the scale and default to their
observed counterparts if not specified.

transpose specifies that variables are observers (coders, judges,
raters, ...) and observations are units of anlysis (subjects, objects,
...). This option does not affect the dataset itself. The by prefix,
if, and in qualifiers apply to the original structure of the
dataset. See xpose to interchange observations and variables in the
dataset.

bootstrap[(boot_options)] bootstraps the
distribution of Krippendorff's alpha to obtain confidence intervals
and probabilities to fail to reach a required minimum alpha. boot_options
are

reps(#) specifies the number of bootstrap replications and
defaults to 20,000.

level(#) sets the confidence level. Default is
level().

minalpha(numlist) bootstraps the probabilities to fail to
reach a minimum alpha of numlist.

seed(#) sets the random-number seed. This option may not be
combined with by. If by is specified use set seed
instead.

[no]dots[(#)] prints a dot each #
replication, where # defaults to max(1, floor(reps/50)).

Example

. clear

. inp u1 u2 u3 u4 u5 u6 u7 u8 u9 u10 u11 u12

1 2 3 3 2 1 4 1 2 . . .

1 2 3 3 2 2 4 1 2 5 . 3

. 3 3 3 2 3 4 2 2 5 1 .

1 2 3 3 2 4 4 1 2 5 1 .

end

. kalpha u1-u12

. kalpha u1-u12 , scale(ordinal)

. kalpha u1-u12 , scale(nominal) bootstrap(minalpha(0.667 0.8))

Saved results

kalpha saves the following in r():

Scalars

r(kalpha) Krippendorff's alpha coefficient

r(observers) number of observers (coders, judges, raters, ...)

r(units) number of units (with pairable values)

r(n) number of pairable values (n_..)

r(Do) observed disagreement

r(De) expected disagreement

Macros

r(metric) metric (level of measurement)

Matrices

r(coin) coincidence matrix

r(delta2) delta matrix

r(uniqv) distinct values (numeric variables only)

With the bootstrap option kalpha additionally returns

Scalars

r(level) confidence level

r(reps) number of replications

r(ci_lb) lower bound of confidence interval

r(ci_ub) upper bound of confidence interval

Matrices

r(q) probabilities to fail to reach minimum alpha

If version is set to 15.1 (or lower), kalpha additionally
returns

Matrices

r(rel) reliability data matrix (numeric variables only)

r(vbu) values-by-units matrix

r(rsum) row sums (n_.c)

r(csum) column sums (n_u.)

References

Gwet, Kilem L. (2015). Standard Error of Krippendorff's Alpha
Coeffcient. K. Gwet's Inter-Rater Reliability Blog.
http://inter-rater-reliability.blogspot.de/2015/08/standard-error-of-krippendorffs-alpha.html

Hayes, Andrew F., Krippendorff, Klaus (2007). Answering the call
for a standard reliability measure for coding data.
Communication Methods and Measures, 1, 77-89.

Krippendorff, Klaus (2013a). Computing Krippendorff's
Alpha-Reliability.
http://www.asc.upenn.edu/usr/krippendorff/mwebreliability5.pdf

Krippendorff, Klaus (2013b). Algorithm for bootstrapping a
distribution of c_a.
http://www.asc.upenn.edu/usr/krippendorff/Bootstrapping%20Revised(5).pdf

Krippendorff, Klaus (2011). Agreement and Information in the Reliability
of Coding. Communication Methods and Measures, 5(2), 93-112.
http://dx.doi.org/10.1080/19312458.2011.568376

Krippendorff, Klaus (2004). Reliability in Content Analysis: Some
common Misconceptions and Recommendations.
Human Communication Research 30, 3, 411-433.

Zapf, Antonia, Castell, Stefanie, Morawietz, Lars, and Karch, André
(2016). Measuring inter-rater reliability for nominal data which
coeffcients and confidence intervals are
appropriate? BMC Medical Research Methodology, 16:93.

Acknowledgments

We are grateful to Klaus Krippendorff and Andrew Hayes for
clarifying questions about the computation of reliability for only
one unit of analysis.

Authors

Daniel Klein, INCHER-Kassel, University of Kassel, klein.daniel.81@gmail.com

Rafael Reckmann, University of Kassel, rafael.reckmann@hotmail.com

Also see

Online: kappa, icc, spearman, egen

if installed: krippalpha, kappaetc