help kappaetc_choosing (SJ18-4: st0544)

Choosing an appropriate method of analysis

The table below is intended to assist in choosing an appropriate method for
analyzing agreement or reliability given the data level of measurement
(compare Gwet [2014, 24]).

Data/ratings Method Command

categorical unweighted agreement kappaetc
nominal coefficients

ordinal agreement coefficients kappaetc, wgt(ordinal)
with ordinal weights

interval agreement coefficients kappaetc, wgt(linear)
(predetermined ratings) with linear, quadratic, kappaetc, wgt(quadratic)
or radical weights kappaetc, wgt(radical)

ratio agreement coefficients kappaetc, wgt()
(predetermined ratings) with any weights

interval or ratio intraclass correlation kappaetc, icc()
coefficients

interval or ratio intraclass correlation kappaetc, icc() id()
(intrarater reliability) coefficients

Krippendoff's (2011, 2013) metric difference functions

Krippendorff (2011, 2013) uses the term "metric difference functions" for the
weights that account for partial agreement or disagreement implied by the
data's level of measurement. The weights that correspond to the respective
metric difference functions are summarized below.

Metric Corresponding
difference function weight Command

categorical none kappaetc
nominal identity kappaetc, wgt(identity)

ordinal ordinal kappaetc, wgt(ordinal, krippendorff)

interval quadratic kappaetc, wgt(quadratic)

ratio ratio kappaetc, wgt(ratio)

circular circular kappaetc, wgt(circular)

bipolar bipolar kappaetc, wgt(bipolar)

References

Gwet, K. L. 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg,
MD: Advanced Analytics.

Krippendorff, K. 2011. Computing Krippendorff's alpha-reliability.
https://repository.upenn.edu/asc_papers/43/.

------. 2013. Content Analysis. An Introduction to Its Methodology. 3rd
ed. Thousand Oaks, CA: Sage.

Author

Daniel Klein

International Centre for Higher Education Research Kassel

Kassel, Germany

klein@incher.uni-kassel.de

Also see

Article: Stata Journal, volume 18, number 4: st0544

Help: kappaetc (if installed)