In 1960, Cohen introduced the kappa coefficient to measure chance-corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence-interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them. RESUME C'est en 1960 que Cohen a propost I'emploi du coefficient kappa comme outil de mesure de I'accord entre deux tvaluateurs exprimant leur jugement au moyen d'une Cchelle nominale. De nombreuses gentralisations de cette mesure d'accord ont Ct C proposies depuis lors. Les auteurs jettent ici un regard critique sur nombre de ces travaux traitant du cas ou I'Cchelle de rtponse est soit nominale, soit ordinale. Les principales approches statistiques sont passCes en revue, les modkles sous-jacents sont dicrits et caractCrisCs, et les problkmes liCs i I'estimation ponctuelle ou par intervalle sont abordCs. L'accent est m i s sur diffkrents scknarios concrets et sur des schtmas exp6rimentaux qui sous-tendent I'emploi de ces mesures et les relations existant entre elles.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.