A new coefficient of interrater agreement: The challenge of highly unequal category proportions.

Oest, Rutger van

doi:10.1037/met0000183

Cited by 10 publications

(24 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the scenario's true chance-corrected agreement, we take either a moderate value or a high value: I r 2 ¼ :49 (i.e., I r ¼ :70) or I r 2 ¼ :81 (i.e., I r ¼ :90). These values for chance-corrected agreement are the same as in Van Oest (2019) and close to the values for Cohen's kappa in a simulation study by De Raadt et al (2019). Following Van Oest's strategy to keep the number of scenarios manageable, we consider only the smallest number of categories given the type of data.…”

Section: Simulationsupporting

confidence: 67%

“…Because Îw 2 in Equations 4 and 5 corresponds to the square of the estimated probability of accurate judgment, we obtain an interpretation: Chance-corrected (weighted) agreement coefficients estimate the probability that both raters in a pair assign an item to its correct category without guessing. Whereas Van Oest (2019) showed this result for dichotomous agreement coefficients, we extend it to weighted agreement with arbitrary weight matrices W ¼ ðw c;c Þ.…”

Section: Chance-corrected Weighted Agreementmentioning

confidence: 60%

“…Following Van Oest (2019), we use Bayesian updating to estimate the unobserved category proportions ðp 1 ; . .…”

Section: Generalized Bayesian Coefficientmentioning

confidence: 99%

“…Similar to Broemeling (2001), Van Oest (2019) used a (uniform) Dirichlet distribution to capture uncertainty in the prior beliefs. However, Van Oest updated category probabilities, whereas Broemeling updated the probabilities of category combinations obtained from all raters.…”

mentioning

confidence: 99%

“…Recently, Van Oest (2019) proposed a Bayesian model-based approach to correcting for chance agreement that encompasses these two approaches and offers a new “hybrid” approach and a new coefficient, called the uniform prior coefficient. This approach begins with the prior belief that all categories have equal probabilities and then updates this belief using the observed data, where the amount of updating increases with the number of items in the sample true(Ntrue).…”

mentioning

confidence: 99%

See 4 more Smart Citations

Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement.

Oest¹,

Girard²

2021

Psychological Methods

Self Cite

View full text Add to dashboard Cite

Van Oest (2019) developed a framework to assess interrater agreement for nominal categories and complete data. We generalize this framework to all four situations of nominal or ordinal categories and complete or incomplete data. The mathematical solution yields a chance-corrected agreement coefficient that accommodates any weighting scheme for penalizing rater disagreements and any number of raters and categories. By incorporating Bayesian estimates of the category proportions, the generalized coefficient also captures situations in which raters classify only subsets of items; that is, incomplete data. Furthermore, this coefficient encompasses existing chance-corrected agreement coefficients: the S-coefficient, Scott's pi, Fleiss' kappa, and Van Oest's uniform prior coefficient, all augmented with a weighting scheme and the option of incomplete data. We use simulation to compare these nested coefficients. The uniform prior coefficient tends to perform best, in particular, if one category has a much larger proportion than others. The gap with Scott's pi and Fleiss' kappa widens if the weighting scheme becomes more lenient to small disagreements and often if more item classifications are missing; missingness biases play a moderating role. The uniform prior coefficient often performs much better than the S-coefficient, but the S-coefficient sometimes performs best for small samples, missing data, and lenient weighting schemes. The generalized framework implies a new interpretation of chance-corrected weighted agreement coefficients: These coefficients estimate the probability that both raters in a pair assign an item to its correct category without guessing. Whereas Van Oest showed this interpretation for unweighted agreement, we generalize to weighted agreement.

show abstract

Section: Simulationsupporting

confidence: 67%

Section: Chance-corrected Weighted Agreementmentioning

confidence: 60%

“…Following Van Oest (2019), we use Bayesian updating to estimate the unobserved category proportions ðp 1 ; . .…”

Section: Generalized Bayesian Coefficientmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement.

Oest¹,

Girard²

2021

Psychological Methods

Self Cite

View full text Add to dashboard Cite

show abstract

The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement

Oest

2022

Psychometrika

Self Cite

View full text Add to dashboard Cite

We consider the dependence of a broad class of chance-corrected weighted agreement coefficients on the weighting scheme that penalizes rater disagreements. The considered class encompasses many existing coefficients with any number of raters, and one real-valued power parameter defines the weighting scheme that includes linear, quadratic, identity, and radical weights. We obtain the first-order and second-order derivatives of the coefficients with respect to the power parameter and decompose them into components corresponding to all pairs of different category distances. Each component compares its two distances in terms of the ratio of observed to expected-by-chance frequency. A larger ratio for the smaller distance than the larger distance contributes to a positive relationship between the power parameter and the coefficient value; the opposite contributes to a negative relationship. We provide necessary and sufficient conditions for the coefficient value to increase or decrease and the relationship to intensify or weaken as the power parameter increases. We use the first-order and second-order derivatives for corresponding measurement. Furthermore, we show how these two derivatives allow other researchers to obtain quite accurate estimates of the coefficient value for unreported values of the power parameter, even without access to the original data.

show abstract

Measuring Agreement Using Guessing Models and Knowledge Coefficients

Moss

2023

Psychometrika

View full text Add to dashboard Cite

Several measures of agreement, such as the Perreault–Leigh coefficient, the $$\textsc {AC}_{1}$$ AC 1 , and the recent coefficient of van Oest, are based on explicit models of how judges make their ratings. To handle such measures of agreement under a common umbrella, we propose a class of models called guessing models, which contains most models of how judges make their ratings. Every guessing model have an associated measure of agreement we call the knowledge coefficient. Under certain assumptions on the guessing models, the knowledge coefficient will be equal to the multi-rater Cohen’s kappa, Fleiss’ kappa, the Brennan–Prediger coefficient, or other less-established measures of agreement. We provide several sample estimators of the knowledge coefficient, valid under varying assumptions, and their asymptotic distributions. After a sensitivity analysis and a simulation study of confidence intervals, we find that the Brennan–Prediger coefficient typically outperforms the others, with much better coverage under unfavorable circumstances.

show abstract

A new coefficient of interrater agreement: The challenge of highly unequal category proportions.

Cited by 10 publications

References 39 publications

Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement.

Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement.

The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement

Measuring Agreement Using Guessing Models and Knowledge Coefficients

Contact Info

Product

Resources

About