help kappaetc, icc() (SJ18-4: st0544)

Title

kappaetc, icc() -- Interrater and intrarater
reliability

Syntax

One-way random-effects model, intraclass correlation coefficients

kappaetc
varname1
varname2
[varname3 ...]
[if] [in], icc(oneway) [options]

Two-way random-effects model, intraclass correlation coefficients

kappaetc
varname1
varname2
[varname3 ...]
[if] [in], icc(random) [options]

Two-way mixed-effects model, intraclass correlation coefficients

kappaetc
varname1
varname2
[varname3 ...]
[if] [in], icc(mixed) [options]

* icc(model [, model_options]) fit
model; see Options

id(varname) specify subject identifier for
repeated measurements

listwise exclude observations with missing ratings

balanced keep only subjects with the same number of repeated
measurements

level(#) set confidence level; default is
level()

testvalue(#) test whether intraclass correlation coefficients
are equal to #; default is testvalue(0)

stddeviations show random effects and error effect as standard
deviations; the default

variance show random effects and error effect as variances

nociclipped do not clip confidence intervals at 0 and 1

noheader suppress output header

notable suppress coefficient table

format_options control column formats

* icc() is required.

by is allowed; see [D] by.

Description

kappaetc with the icc() option estimates intraclass correlation
coefficients as a measure of interrater and intrarater reliability. The
command implements one-way random-effects, two-way random-effects and
mixed-effects models, optionally with repeated measurements.

All models are implemented using formulas for unbalanced designs given in Gwet
(2014). Intraclass correlation coefficients are estimated in the presence of
missing values; test statistics and confidence intervals, however, are based
on methods for complete data.

kappaetc with the icc() option assumes that observations are
subjects (units of analysis) and variables contain the ratings by raters
(coders, judges, observers). Subjects with m repeated measurements are
represented by m observations in the dataset.

Options

icc(model [, model_options]) specifies the model
to be fit. icc() is required. The available models and
model_options are described below.

oneway fits the one-way random-effects model (model 1A). This model
assumes that each subject is rated by a different group of raters. Thus, the
subject is the only (random) factor. Intrarater reliability cannot be
assessed. The model_option b is allowed and requests model 1B to
be fit. Here the rater is the single (random) factor, and each of the raters
may rate a different set of subjects. This latter model is seldom used and
cannot be used to estimate interrater reliability.

random fits the two-way random-effects model (model 2). Here each
subject is rated by the same group of raters. Subjects are assumed to be
randomly selected from a universe of subjects, and raters are assumed to be
drawn from the rater population. The model_option blend is allowed
and specifies that the subject-rater interaction be blended together with the
error variance; this is relevant only if some subjects are rated repeatedly by
the same rater.

mixed fits the two-way mixed-effects model (model 3). In this model,
each subject is rated by the same group of raters. Subjects are assumed to be
randomly selected from a universe of subjects. Raters are assumed to be the
only ones of interest and do not represent any larger rater population. The
model_option blend is allowed and specifies that the subject-rater
interaction be blended together with the error variance; this is relevant only
if some subjects are rated repeatedly by the same rater.

id(varname) declares varname the subject identifier when subjects
are measured repeatedly. If specified with a two-way model, interrater and
intrarater reliability is estimated.

listwise specifies that observations with missing ratings be excluded
from the analysis. Do not confuse observations with subjects in the case of
repeated measurements. This option does not create a balanced design; for
that, see option balanced below. casewise is a synonym for
listwise.

balanced restricts the estimation sample to subjects with the same
number of repeated measures. If the maximum number of repeated measurements is
m, then all subjects that have been measured less than m times are excluded.
Note that this option does not exclude observations with missing values; for
that, see option listwise above.

level(#) specifies the confidence level, as a percentage, for confidence
intervals. The default is level().

testvalue(#) tests whether intraclass correlation coefficients are equal
to #. The default is testvalue(0). One-sided F tests are
performed.

stddeviations displays the random effects and error effect as standard
deviations. This is the default.

variance displays the random effects and error effect as variances.

nociclipped reports confidence intervals as estimated. The default is
to restrict confidence limits to fall into the range of 0 < # < 1.

noheader suppresses the report about the number of subjects, ratings per
subject, and rating categories. Only the coefficient table is displayed.

notable suppresses the display of the coefficient table.

format_options are the same as with
kappaetc.

Examples

Examples are drawn from icc and [R] icc

. webuse judges

. reshape wide rating, i(target) j(judge)

One-way random-effects model

. kappaetc rating1-rating4, icc(oneway)

Two-way random-effects model

. kappaetc rating1-rating4, icc(random)

Stored results

kappaetc with the icc() option stores the following in r():

Scalars

r(N) number of subjects

r(r) number of raters

r(r_min) minimum number of ratings per subject

r(r_avg) average number of ratings per subject

r(r_max) maximum number of ratings per subject

r(M) number of observed measurements

r(M_missing) number of missing measurements

r(m_min) minimum number of replicates per subject

r(m_avg) average number of replicates per subject

r(m_max) maximum number of replicates per subject

r(icc) intraclass correlation coefficient
(interrater reliability)

r(icc_a) intraclass correlation coefficient
(intrarater reliability)

r(sigma2_s) subject variance

r(sigma2_r) rater variance

r(sigma2_sr) subject-rater interaction variance

r(sigma2_e) error variance

r(MSS) mean squares between subjects

r(MSR) mean squares between raters

r(MSI) subject-rater interaction mean squares

r(MSE) error mean squares

r(level) confidence level

r(icc_df_2) denominator degrees of freedom for intracluster correlation coefficient (ICC)

r(icc_df_1) numerator degrees of freedom for ICC

r(icc_F) F statistic for ICC

r(icc_a_df_2) denominator degrees of freedom for ICCa
(repeated measurements only)

r(icc_a_df_1) numerator degrees of freedom for ICCa
(repeated measurements only)

r(icc_a_F) F statistic for ICCa
(repeated measurements only)

r(has_sr) whether subject-rater interaction has been
estimated

Macros

r(cmd) kappaetc

r(cmd2) icc

r(model) fit model (oneway, random,
or mixed)

r(model_number) fit model number (1, 1B,
2, or 3)

Matrices

r(table) information from the coefficient table

Reference

Gwet, K. L. 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed.
Gaithersburg, MD: Advanced Analytics.

Author

Daniel Klein

International Centre for Higher Education Research Kassel

Kassel, Germany

klein@incher.uni-kassel.de

Also see

Article: Stata Journal, volume 18, number 4: st0544

Help: [R] kappa, [R] icc, kappaetc (if installed)