A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research

Abstract

Objective: Intraclass correlation coefficient (ICC) is a widely used reliability index in testretest, intrarater, and interrater reliability analyses. This article introduces the basic concept of ICC in the content of reliability analysis. Discussion for Researchers: There are 10 forms of ICCs. Because each form involves distinct assumptions in their calculation and will lead to different interpretations, researchers should explicitly specify the ICC form they used in their calculation. A thorough review of the research design is needed in selecting the appropriate form of ICC to evaluate reliability. The best practice of reporting ICC should include software information, "model," "type," and "definition" selections. Discussion for Readers: When coming across an article that includes ICC, readers should first check whether information about the ICC form has been reported and if an appropriate ICC form was used. Based on the 95% confident interval of the ICC estimate, values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 are indicative of poor, moderate, good, and excellent reliability, respectively. Conclusion: This article provides a practical guideline for clinical researchers to choose the correct form of ICC and suggests the best practice of reporting ICC parameters in scientific publications. This article also gives readers an appreciation for what to look for when coming across ICC while reading an article.

Efficient identification of mental health problems in refugees in Germany: the Refugee Health Screener

“…above measurement error. The reliability was calculated with intraclass correlation coefficients (ICC) estimates based on a mean-rating, absolute-agreement, two-way random effects model (Koo & Li, 2016; McGraw & Wong, 1996). ICC estimates were interpreted as <.5 poor, .5–.75 moderate, .75–.90 good, and >.90 excellent reliability (Koo & Li, 2016).…”

Using New and Innovative Technologies to Assess Clinical Stage in Early Intervention Youth Mental Health Services: Evaluation Study (Preprint)

“…Clinicians also completed the Social and Occupational Functioning Assessment Scale (SOFAS) [46] and the Clinical Global Impression Scale-Severity (CGI-S) [47] [49]. For the continuous variables (SOFAS and CGI-S), Intraclass Correlation Coefficient (ICC) was used to calculate agreement between offline and online clinicians [50], and interpretations were based on a 95% confidence interval where estimates less than 0.5 reflect poor agreement, ICC=0.5-0.75 moderate agreement, ICC=0.75-0.90 good agreement and greater than 0.9 excellent agreement [51].…”

Face-to-Face Versus Video Assessment of Facial Paralysis: Implications for Telemedicine (Preprint)

“…ICC (1,1) was selected because each participant was assessed by a different set of assessors. For each participant, scores given by three randomly selected assessors were used to compute the ICC, hence the rater was a random effect [38,39]. Portney and Watkins [38] noted that "ICC ranges between 0.00 and 1.00, with values closer to 1.00 representing stronger reliability", but there were "no standard values for acceptable reliability using the ICC", and that the researcher should determine the level of reliability needed to justify the use of the tool being assessed.…”

“…Based on the study by Banks et al [31], we defined reliability as poor for ICC values less than 0.40, fair for values between 0.40 and 0.59, good for values between 0.60 and 0.74, and excellent for values 0.75 and above. The 95% CI indicates that there is 95% chance that any score would lie within the range [39].…”