Weighted kappa as a function of unweighted kappas

Moradzadeh, N; Ganjali, Mojtaba; Baghfalaki, Taban

doi:10.1080/03610918.2015.1105975

Cited by 4 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The result specifies that symmetric kappa corresponding to a c × c table with c ≥ 3 categories can be written as a function of the unweighted kappa corresponding to the same table and the c(c − 1)/2 distinct unweighted kappas associated with the (c − 1) × (c − 1) tables that are obtained by combining two categories. Moradzadeh et al (2017) also showed that weighted kappa can be written as a function of various unweighted kappas. Their result requires that pairs of categories that are the same number of steps apart have identical weights.…”

Section: Resultsmentioning

confidence: 99%

“…In other words, formula (17) shows that symmetric kappa is not only a function, but also a weighted average of the three unweighted kappas if and only if v < 1 2 . Formulas (16) and (17) shows that the result in Moradzadeh et al (2017) is a special case of Theorem 3.1 in Section 3 if the rating system has precisely three categories. If the system has four or more categories the result in Moradzadeh et al (2017) and Theorem 3.1 are different.…”

Section: The 3 × 3 Casementioning

confidence: 99%

“…In other words, formula (16) shows that symmetric kappa is not only a function, but also a weighted average of the four unweighted kappas if and only if w < 1. Moradzadeh et al (2017) considered the case where w 12 = v = w 23 and w 13 = 0. In this case, symmetric kappa is given by…”

Section: The 3 × 3 Casementioning

confidence: 99%

“…A completely new type of result was presented in Moradzadeh et al (2017), or MGB for short. These authors showed that certain weighted kappas are functions of unweighted kappas.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Symmetric kappa as a function of unweighted kappas

Warrens¹

2017

Communications in Statistics - Simulation and Computation

View full text Add to dashboard Cite

show abstract

Section: Resultsmentioning

confidence: 99%

Section: The 3 × 3 Casementioning

confidence: 99%

Section: The 3 × 3 Casementioning

confidence: 99%

“…A completely new type of result was presented in Moradzadeh et al (2017), or MGB for short. These authors showed that certain weighted kappas are functions of unweighted kappas.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Symmetric kappa as a function of unweighted kappas

Warrens¹

2017

Communications in Statistics - Simulation and Computation

View full text Add to dashboard Cite

show abstract

“…In social, behavioral and biomedical sciences kappa coefficients are commonly used for quantifying agreement between two classifications with identical categories (Vanbelle 2016;Warrens 2014Warrens , 2017. Agreement between classifications with nominal categories is usually assessed with Cohen's kappa (Cohen 1960;Brennan and Prediger 1981;Maclure and Willett 1987;Kundel and Polansky 2003;Hsu and Field 2003;Conger 2017), whereas agreement between classifications with ordinal categories is commonly assessed with weighted kappa coefficients (Cohen 1968;Vanbelle and Albert 2009;Warrens 2011Warrens , 2012Yang and Zhou 2015;Vanbelle 2016;Moradzadeh et al 2017). These commonly used kappa coefficients have been extended in various directions.…”

Section: Introductionmentioning

confidence: 99%

Kappa coefficients for dichotomous-nominal classifications

Warrens

2020

Adv Data Anal Classif

View full text Add to dashboard Cite

Two types of nominal classifications are distinguished, namely regular nominal classifications and dichotomous-nominal classifications. The first type does not include an 'absence' category (for example, no disorder), whereas the second type does include an 'absence' category. Cohen's unweighted kappa can be used to quantify agreement between two regular nominal classifications with the same categories, but there are no coefficients for assessing agreement between two dichotomous-nominal classifications. Kappa coefficients for dichotomous-nominal classifications with identical categories are defined. All coefficients proposed belong to a one-parameter family. It is studied how the coefficients for dichotomous-nominal classifications are related and if the values of the coefficients depend on the number of categories. It turns out that the values of the new kappa coefficients can be strictly ordered in precisely two ways. The orderings suggest that the new coefficients are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for the new coefficients that tend to produce higher values.

show abstract

A Comparison of Reliability Coefficients for Ordinal Rating Scales

et al. 2021

View full text Add to dashboard Cite

Kappa coefficients are commonly used for quantifying reliability on a categorical scale, whereas correlation coefficients are commonly applied to assess reliability on an interval scale. Both types of coefficients can be used to assess the reliability of ordinal rating scales. In this study, we compare seven reliability coefficients for ordinal rating scales: the kappa coefficients included are Cohen’s kappa, linearly weighted kappa, and quadratically weighted kappa; the correlation coefficients included are intraclass correlation ICC(3,1), Pearson’s correlation, Spearman’s rho, and Kendall’s tau-b. The primary goal is to provide a thorough understanding of these coefficients such that the applied researcher can make a sensible choice for ordinal rating scales. A second aim is to find out whether the choice of the coefficient matters. We studied to what extent we reach the same conclusions about inter-rater reliability with different coefficients, and to what extent the coefficients measure agreement in a similar way, using analytic methods, and simulated and empirical data. Using analytical methods, it is shown that differences between quadratic kappa and the Pearson and intraclass correlations increase if agreement becomes larger. Differences between the three coefficients are generally small if differences between rater means and variances are small. Furthermore, using simulated and empirical data, it is shown that differences between all reliability coefficients tend to increase if agreement between the raters increases. Moreover, for the data in this study, the same conclusion about inter-rater reliability was reached in virtually all cases with the four correlation coefficients. In addition, using quadratically weighted kappa, we reached a similar conclusion as with any correlation coefficient a great number of times. Hence, for the data in this study, it does not really matter which of these five coefficients is used. Moreover, the four correlation coefficients and quadratically weighted kappa tend to measure agreement in a similar way: their values are very highly correlated for the data in this study.

show abstract

Weighted kappa as a function of unweighted kappas

Cited by 4 publications

References 31 publications

Symmetric kappa as a function of unweighted kappas

Symmetric kappa as a function of unweighted kappas

Kappa coefficients for dichotomous-nominal classifications

A Comparison of Reliability Coefficients for Ordinal Rating Scales

Contact Info

Product

Resources

About