Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial

Abstract

Many research designs require the assessment of inter-rater reliability (IRR) to demonstrate consistency among observational ratings provided by multiple coders. However, many studies use incorrect statistical procedures, fail to fully report the information necessary to interpret their results, or do not address how IRR affects the power of their subsequent analyses for hypothesis testing. This paper provides an overview of methodological issues related to the assessment of IRR with a focus on study design, selection of appropriate statistics, and the computation, interpretation, and reporting of some commonly-used IRR statistics. Computational examples include SPSS and R syntax for computing Cohen's kappa and intra-class correlations to assess IRR.The assessment of inter-rater reliability (IRR, also called inter-rater agreement) is often necessary for research designs where data are collected through ratings provided by trained or untrained coders. However, many studies use incorrect statistical analyses to compute IRR, misinterpret the results from IRR analyses, or fail to consider the implications that IRR estimates have on statistical power for subsequent analyses. This paper will provide an overview of methodological issues related to the assessment of IRR, including aspects of study design, selection and computation of appropriate IRR statistics, and interpreting and reporting results. Computational examples include SPSS and R syntax for computing Cohen's kappa for nominal variables and intraclass correlations (ICCs) for ordinal, interval, and ratio variables. Although it is beyond the scope of the current paper to provide a comprehensive review of the many IRR statistics that are available, references will be provided to other IRR statistics suitable for designs not covered in this tutorial.

Patient-Specific Anatomical and Functional Parameters Provide New Insights into the Pathomechanism of Cam FAI

“…Intrarater considered single measures whereas interrater considered average measures, both seeking absolute agreement [19,42]. The intra-and interrater anatomical observations were in strong to near-perfect agreement (0.847 B ICC Obs1 B 0.987; 0.867 B ICC Obs2 B 0.967; 0.703 B ICC Obs1-2 B 0.886; Table 1).…”

Interest in a Twitter-delivered weight loss program among women of childbearing age

“…Then, two coders (MEW and RSX) independently coded each response as fitting one of more of these themes (i.e., each response was coded as fitting or not fitting each of the themes). We calculated the percent agreement on coding and kappas for the coding of each theme [31]. For the survey responses with discrepantly coded themes, the coders reached consensus through discussion.…”

Investigating Measures of Intensity During a Structured Upper Limb Exercise Program in Stroke Rehabilitation: An Exploratory Study

“…Observed repetitions from both sets of data collection forms were extracted into SPSS to analyse inter-rater reliability (IRR). As number of repetitions were ordinal level data the interclass correlation coefficient was used to assess IRR as advised by Hallgren 31 . An ICC of 0.968 indicated substantial reliability between raters.…”