*! version 0.1.0 01nov2017

Title

ira -- Stata module to calculate rwg and related interrater agreement indices

Syntax

ira
judge_id
rating
[if]
[in]
[, item(item_id) group(group_id) options(#) distribution(#)]

item variable that identifies scale items; if none is specified, ira will assume a single-item scale

group variable that identifies groups; if none is specified, ira will assume a single group

options(#) number of response options (anchors) on scale;
default is options(7)

distribution(#) variance of user-specified null distribution for rwg(j)

Description

ira calculates the within-group interrater agreement indices rwg(j), r*wg(j), r'wg(j), awg(j), and AD(j).

ira calculates these indices for all groups in the data and reports mean, median, and range (absolute difference between maximum and minimum) across groups for each index.

To do so, ira needs the user to specify the following variables:

- judge_id: identifier for the judge (rater); must be unique within a group

- rating: judges' ratings (scores)

- item_id: identifier for the scale item (if a multiple-item scale was employed)

- group_id: identifier for the group (target) (if there is more than one)

The dataset must be in "long" format. Use the reshape command if needed.

Notes on ratings:

- Ratings must be values between 1 and the number of scale options specified through options(#) (ira assumes that the scale's lowest response option is 1).

- Ratings should be numeric (although ira will make an attempt to de-string if needed). Ratings need not be integer values. However, if they are not, the interpretation of some indices may change.

- All judges of a group must provide ratings for all items.

ira can also provide results for cases in which there is only one group or provide results separately by group. There are at least three ways of doings this, depending on how the user's dataset is set up.

1) If the dataset only includes data for a single group, simply omit the group(group_id) option. All indices reported by ira will pertain to the one group.

2) If the dataset comprises data for multiple groups, but individual indices for a single group are desired, ira can be invoked using the "if" functionality to indicate which group should be analyzed. If the grouping variable is called my_group_id and results are desired for group 3, the user might specify:

. ira judge rating if my_group_id == 3

3) If the dataset comprises data for multiple groups and individual indices for all groups are desired, ira can be used with "by":

. by my_group_id: ira judge rating

Reported indices

ira produces the following indices:

rwg(j)

ira calculates James et al.'s (1984, 1993) rwg(j) indices using various null distributions: uniform, skewed, triangular, and normal. Results for non-uniform distributions are only calculated if the specified number of scale items is between 5 and 11. Variances for non-uniform null distributions are taken from LeBreton & Senter (2008).

If the distribution(#) option is specified, ira calculates an additional rwg index using the supplied parameter as the variance of the null distribution.

Finn (1970) does not specify a range for rwg but explains that unity indicates perfect agreement and zero indicates no agreement (i.e., agreement equal to the null distribution). Following James et al. (1984), ira sets rwg(j) values for groups to zero if they exceed unity or are negative. If this happens, ira notifies the user. Possible reasons for such values include sampling error (James et al., 1984) and inappropriate specification of the null distribution.

r*wg(j)

Lindell et al. (1999) propose r*wg(j) as an improvement over rwg(j) that does not increase as the number of response options in the scale increases.

Interpretation is similar to rwg(j). Unlike rwg(j), negative values are not set to zero.

Lindell et al. (1999) suggest several indices. ira implements what they refer to as "Index C."

r'wg(j)

Lindell (2001) proposes further alternatives to rwg(j). ira calculates all four proposed variants.

Interpretation is similar to rwg(j).

awg(j)

ira calculates the awg(j) index as proposed by Brown & Hauenstein (2005).

The measure attempts to address limitations of rwg(j), namely that the magnitude of the measure depends on sample size and the used scale.

awg(j) is an analogue to Cohen's kappa. A value of positive unity indicates perfect agreement, given the group mean. A value of negative unity indicates maximum disagreement, given the group mean.

awg(j) canot be interpreted if a group's mean is extremely low or high. In such cases, ira will issue a warning and omit the case from the calculation of mean, median, and range across groups.

AD(j)

ira follows Burke et al. (1999) and calculates two variants: Average deviation of the mean (ADM(j)) and average deviation of the median (ADMd(j)).

Remarks

For rwg-type measures, values >.70 are conventionally taken to indicate acceptable interrater agreement. See Harvey & Hollander (2004) for a critical discussion. See LeBreton & Senter (2008) and O'Neill (2017) for further information on cut-off values for rwg(j) and other measures.

ira does not calculate statistical significance of the interrater agreement measures. See Dunlap et al. (2003) and Smith-Crowe et al. (2014) for further information and critical values.

For general reference regarding various interrater agreement indices, see LeBreton & Senter (2008) and O'Neill (2017).

Citation

ira is not an official Stata command. It is a free contribution to the research community. If you use it, please cite it as:

Graf-Vlachy, L. 2017. ira: Stata module to calculate rwg and related interrater agreement indices, http://www.repec.org/bocode/i/ira.html.

Stored results

ira stores the following in r():

Scalars

r(N) number of observations

r(J) number of items

r(groups) number of groups

r(A) number of response options

r(rwg_unif_mean) mean of rwg(j) across groups (uniform null distribution)

r(rwg_unif_median) median of rwg(j) across groups (uniform null distribution)

r(rwg_unif_range) range of rwg(j) across groups (uniform null distribution)

r(rwg_slight_skew_mean) mean of rwg(j) across groups (slightly skewed null distribution)

r(rwg_slight_skew_median) median of rwg(j) across groups (slightly skewed null distribution)

r(rwg_slight_skew_range) range of rwg(j) across groups (slightly skewed null distribution)

r(rwg_mod_skew_mean) mean of rwg(j) across groups (moderately skewed null distribution)

r(rwg_mod_skew_median) median of rwg(j) across groups (moderately skewed null distribution)

r(rwg_mod_skew_range) range of rwg(j) across groups (moderately skewed null distribution)

r(rwg_heavy_skew_mean) mean of rwg(j) across groups (heavily skewed null distribution)

r(rwg_heavy_skew_median) median of rwg(j) across groups (heavily skewed null distribution)

r(rwg_heavy_skew_range) range of rwg(j) across groups (heavily skewed null distribution)

r(rwg_tri_mean) mean of rwg(j) across groups (triangular null distribution)

r(rwg_tri_median) median of rwg(j) across groups (triangular null distribution)

r(rwg_tri_range) range of rwg(j) across groups (triangular null distribution)

r(rwg_norm_mean) mean of rwg(j) across groups (normal null distribution)

r(rwg_norm_median) median of rwg(j) across groups (normal null distribution)

r(rwg_norm_range) range of rwg(j) across groups (normal null distribution)

r(rwg_custom_mean) mean of rwg(j) across groups (user-specified null distribution)

r(rwg_custom_median) median of rwg(j) across groups (user-specified null distribution)

r(rwg_custom_range) range of rwg(j) across groups (user-specified null distribution)

r(rstarwg_mean) mean of r*wg(j) across groups

r(rstarwg_median) median of r*wg(j) across groups

r(rstarwg_range) range of r*wg(j) across groups

r(rapowgA_mean) mean of r'wg(A) across groups

r(rapowgA_median) median of r'wg(A) across groups

r(rapowgA_range) range of r'wg(A) across groups

r(rapowgB_mean) mean of r'wg(B) across groups

r(rapowgB_median) median of r'wg(B) across groups

r(rapowgB_range) range of r'wg(B) across groups

r(rapowgC_mean) mean of r'wg(C) across groups

r(rapowgC_median) median of r'wg(C) across groups

r(rapowgC_range) range of r'wg(C) across groups

r(rapowgD_mean) mean of r'wg(D) across groups

r(rapowgD_median) median of r'wg(D) across groups

r(rapowgD_range) range of r'wg(D) across groups

r(awg_mean) mean of awg(j) across groups

r(awg_median) median of awg(j) across groups

r(awg_range) range of awg(j) across groups

r(ADM_mean) mean of ADM across groups

r(ADM_median) median of ADM across groups

r(ADM_range) range of ADM across groups

r(ADMd_mean) mean of ADMd across groups

r(ADMd_median) median of ADMd across groups

r(ADMd_range) range of ADMd across groups

Examples

Multiple groups, multi-item five-point scale

. ira judge_id rating, item(item_id) group(group_id) options(5)

Single group, multi-item scale

. ira judge_id rating, item(item_id)

Multiple groups, single-item scale

. ira judge_id rating, group(group_id)

Single group, single-item six-point scale

. ira judge_id rating, options(6)

Single group, single-item scale

. ira judge_id rating

References

Brown, R. D., & Hauenstein, N. M. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8(2), 165-184.

Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2(1), 49-68.

Dunlap, W. P., Burke, M. J., & Smith-Crowe, K. (2003). Accurate tests of statistical significance for rWG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88(2), 356-362.

Finn, R. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30(1), 71-76.

Harvey, R. J., & Holl, E. (2004). Benchmarking rwg interrater agreement indices: Let’s drop the .70 rule-of-thumb. Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology.

James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69(1), 85-98.

James, L. R., Demaree, R. G., & Wolf, G. (1993). r wg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78(2), 306-309.

LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815-852.

Lindell, M. K. (2001). Assessing and testing interrater agreement on a single target using multi-item rating scales. Applied Psychological Measurement, 25(1), 89-99.

Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23(2), 127-135.

O'Neill, T. A. (2017). An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners. Frontiers in Psychology, 8:777.

Smith-Crowe, K., Burke, M. J., Cohen, A., & Doveh, E. (2014). Statistical significance criteria for the rWG and average deviation interrater agreement indices. Journal of Applied Psychology, 99(2), 239-261.

Author

Lorenz Graf-Vlachy

graf-vlachy.com

mail@graf-vlachy.com