*! version 1.0.0 15sep2008
help kappa2

Title

kappa2 -- Generalizations of weighted kappa for incomplete designs

Syntax

Interrater agreement, nonunique raters, variables record ratings for each rater

kappa2 varname1 varname2
varname3 [...] [if] [in] [, options]

Main
tab display table of assessments (available for two raters only)

wgt(wgtid) specify how to weight disagreements; see
Options for alternatives

absolute treat rating categories as absolute

majority the agreement is defined as majority or consensus

jackknife compute jackknifed standard errors and display additional statistics

Dialog box

There is a graphical dialog box from which this command may be used. It can be accessed with the db kappa2 command.

Add this command to the Stata user menu

If you wish to add the command to the Stata menu, please execute the following command (you might add it to you user profile.do that it gets executed every time you start Stata):
window menu append item "stUserStatistics" "&Weighted kappa for incomplete designs" "db kappa2".
Then you must execute
window menu refresh.
This will add the Kappa2 option to the User > Statistics

Description

kappa2 is a generalization of the kappa coefficient to allow using weights and an explicit agreement
definition for multiple raters and incomplete designs. Computations are done using formulae proposed by
Abraira V. In the particular
case of unweighted kappa, kappa2 would reduce to the standard kappa Stata command,
although slight differences could appear because the standard kappa Stata command uses
approximated formulae (see [R] kappa).

kappa2 calculates the kappa-statistic measure of interrater agreement when
there are two ore more unique raters and two or more ratings. It assumes that each
observation is a subject. varname1 contains the ratings
by the first rater, varname2 by the second rater, and so on.

To facilitate the use of this command, its syntax is an extension of the standard kappa command, and hence kappa2
has the same options than kappa but it includes some new ones. This help document copies the standard kappa help
and introduces the novelties.

Options

Main

tab displays a tabulation of the assessments by the two raters.

wgt(wgtid) specifies that wgtid be used to weight disagreements.
You can define your own weights by using kapwgt; wgt() then
specifies the name of the user-defined matrix. For instance, you might define

. kapwgt mine 1 \ .8 1 \ 0 .8 1 \ 0 0 .8 1

and then

. kappa2 rata ratb, wgt(mine)

Also two prerecorded weights are available.

wgt(w) specifies weights 1-|i-j|/(k-1),
where i and j index the rows and columns of the ratings by the two
raters and k is the maximum number of possible ratings.

wgt(w2) specifies weights 1 - {(i-j)/(k-1)}^2.

absolute is relevant only if wgt() is also specified. Option
absolute modifies how i, j, and k are defined and how corresponding
entries are found in a user-defined weighting matrix. When absolute is
not specified, i and j refer to the row and column index, not to the ratings
themselves. Say that the ratings are recorded as {0,1,3,4}. There
are four ratings; k=4, and i and j are still 1, 2, 3, 4 in the formulas above.
Index 2, for instance, corresponds to rating=1. This system is convenient
but can, with some data, lead to difficulties.

When absolute is specified, all ratings must be integers, and they
must be coded from the set {1,2,3,...}. Not all values need be
used; integer values that do not occur are simply assumed to be unobserved.

majority displays majority agreement. In a study with multiple raters,
agreement among raters can be alternatively defined as majority; there is
agreement at an observation if a predefined majority of observers agree; e.g., if there are
seven observers, it is possible to define agreement when at least five (or six) of them agree.
Abraira V, Pérez de Vargas A. Qüestiió (1999) 23: 561-571.

jackknife displays additional statistics. With this option, the program calculates
the jackknife estimate of kappa index, its jackknife standard error and the 95% confidence
interval.

Remarks

You have data on individual patients. There are two raters, and the possible
ratings are 1, 2, 3, and 4, but neither rater ever used rating 3
Here kappa2 would determine that the ratings
are from the set {1,2,4} because those were the only values
observed. kappa2 would expect a user-defined weighting matrix would be 3x3
and, if it were not, kappa2 would issue an error message. In the
formula-based weights, wgt(w) and wgt(w2), the calculation would be
based on i,j = 1,2,3 corresponding to the three observed ratings
{1,2,4}.

Specifying the absolute option would make it clear that the ratings are
1, 2, 3, and 4; it just so happens that rating==3 was never assigned. If a
user-defined weighting matrix were also specified, kappa2 would expect it
to be 4x4 or larger (larger because one can think of the ratings being 1, 2,
3, 4, 5, ... and it just so happens that ratings 5, 6, ... were never
observed, just as rating==3 was not observed). In the formula-based weights,
the calculation would be based on i,j = 1,2,4.

If all conceivable ratings are observed in the data, specifying
absolute makes no difference.

Example: weighted kappa in an incomplete design with multiple raters

Agreement among 6 dermatologists assessing scleroderma using 4 ordinal categories (no scleroderma,
slight scleroderma, moderate scleroderma, atrophic skin).
Abraira V, Pérez de Vargas A. Qüestiió (1999) 23: 561-571.

| med1 med2 med3 med4 med5 med6
-----+

obs1 | 4 . . 4 . 4
obs2 | . . 4 4 2 .
obs3 | . . 3 3 . 3
obs4 | 3 . 3 . 2 .
obs5 | 1 . . . 1 1
obs6 | 1 1 1 . . .
obs7 | . 3 4 . . 3
obs8 | . 1 . . 1 1
obs9 | 2 2 . 3 . .
obs10| . 1 . 1 2 .

. kappa2 med1-med6 (Pairwise agreement)

. kappa2 med1-med6, jackknife (Pairwise agreement with jackknifed estimates)

. kappa2 med1-med6, majority(3) (Majority agreement of 3)

. kappa2 med1-med6, wgt(w) jackknife (weighted agreement with jackknifed estimates)

Saved results

kappa2 save the following in r():

Scalars

r(N) number of subjects

r(prop_o) observed proportion of agreement

r(prop_e) expected proportion of agreement

r(kappa) kappa

In addition, when majority is specified,kappa2 save estimation results in e()

References

Abraira V, Pérez de Vargas A. Generalization of the kappa coefficient for ordinal categorical data,
multiple observers and incomplete designs. Qüestiió (1999) 23: 561-571). http://www.idescat.cat/sort/questiio/questiiopdf/23.3.8.Abraira.pdf

Abraira V. Precisión de las clasificaciones clínicas. Doctoral Thesis. Complutense University of Madrid. Spain. http://www.ucm.es/eprints/4168/

J Lázaro, J Zamora, V Abraira. Implementación de dos generalizaciones del índice kappa. 3ª Reunión Española de Usuarios de Stata Sep 2010: Madrid

Acknowledgment

The development of this macro has been funded by a specific funding action from CIBER
de Edidemiología y Salud Pública (CIBERESP) Spain (PM08/008).

We want to thank Dr. Emparanza and Dr. Pijoan for their valuable comments and for testing the final macro.

Authors

Javier Lázaro, Javier Zamora & Víctor Abraira.
Clinical Biostatistics Unit, Hospital Ramón y Cajal. Madrid. Spain.
CIBER de Edidemiología y Salud Pública (CIBERESP). Spain.

Alexander Zlotnik (GUI, dialog box and minor changes to kappaAux.do)
Department of Electronic Engineering. Universidad Politécnica de Madrid. Spain.

Also see

Manual: [R] kappa