Beyond kappa: A review of interrater agreement measures

Abstract

In 1960, Cohen introduced the kappa coefficient to measure chance-corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence-interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them. RESUME C'est en 1960 que Cohen a propost I'emploi du coefficient kappa comme outil de mesure de I'accord entre deux tvaluateurs exprimant leur jugement au moyen d'une Cchelle nominale. De nombreuses gentralisations de cette mesure d'accord ont Ct C proposies depuis lors. Les auteurs jettent ici un regard critique sur nombre de ces travaux traitant du cas ou I'Cchelle de rtponse est soit nominale, soit ordinale. Les principales approches statistiques sont passCes en revue, les modkles sous-jacents sont dicrits et caractCrisCs, et les problkmes liCs i I'estimation ponctuelle ou par intervalle sont abordCs. L'accent est m i s sur diffkrents scknarios concrets et sur des schtmas exp6rimentaux qui sous-tendent I'emploi de ces mesures et les relations existant entre elles.

An evaluation of non-metric cranial traits used to estimate ancestry in a South African sample

“…The first (CVR) and second (ENL) observer scored these traits on two separate occasions. The strength of agreement between these scoring sessions was calculated using Cohen's Kappa statistic [9]. Agreement was determined to be excellent with a correlation coefficient greater than 0.75, moderate when the correlation coefficient was between 0.40 and 0.74, and poor when it was less than 0.40.…”

Crime seriousness and participation in restorative justice: The role of time elapsed since the offense.

“…In addition, to prevent any reference to differences in the seriousness of the offenses, the seriousness scores were removed from the RMG list for both coders and the offenses in this list were ordered alphabetically based on their law section description. Using Cohen's Kappa formula, the interrater-agreement for the 60 cases that both coders handled was  = .63, p < .001, 95% CI [.49, .76], which is an fair to good level of agreement beyond chance (Banerjee, Capozzoli, McSweeney & Sinha, 1999;Landis & Koch, 1977). Accordingly, the incarceration seriousness scores that corresponded to the offense types from the RMG list were assigned to each mediation case based on the 199 codings of the first coder.…”

Re-evaluation of community of inquiry model with its metacognitive presence construct

“…For instance, the negotiated IRR of the cognitive presence coding scheme ranged from RC=0.6522-0.7957 and K=0.5014-0.6513. This signified that there was a poor agreement (according to [29]) between the two coders. Furthermore, the comments of the coders disclosed the difficulties that they had encountered when following the coding schemes.…”