An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners

Abstract

Applications of interrater agreement (IRA) statistics for Likert scales are plentiful in research and practice. IRA may be implicated in job analysis, performance appraisal, panel interviews, and any other approach to gathering systematic observations. Any rating system involving subject-matter experts can also benefit from IRA as a measure of consensus. Further, IRA is fundamental to aggregation in multilevel research, which is becoming increasingly common in order to address nesting. Although, several technical descriptions of a few specific IRA statistics exist, this paper aims to provide a tractable orientation to common IRA indices to support application. The introductory overview is written with the intent of facilitating contrasts among IRA statistics by critically reviewing equations, interpretations, strengths, and weaknesses. Statistics considered include rwg, rwg*, r′wg, rwg(p), average deviation (AD), awg, standard deviation (Swg), and the coefficient of variation (CVwg). Equations support quick calculation and contrasting of different agreement indices. The article also includes a “quick reference” table and three figures in order to help readers identify how IRA statistics differ and how interpretations of IRA will depend strongly on the statistic employed. A brief consideration of recommended practices involving statistical and practical cutoff standards is presented, and conclusions are offered in light of the current literature.

A scattered CAT: A critical evaluation of the consensual assessment technique for creativity research.

“…**See Myszkowski and Storme (this issue) for a discussion of limitations and alternative test options.Though the tests may seemingly produce roughly similar results, there is not always evidence that this has been checked, and the reasons for choosing one test over another, in the absence of explicit justification, can seem arbitrary or based on convention alone. Likewise, the use of other measures to calculate interrater agreement (IRA) popular in other fields of study, such as Finn's rwg, raises similar issues(O'Neill, 2017). While rwg is not a measure commonly used in CAT studies, it has been used by some (e.g.,Wigert, Reiter-Palmon,…”

The Ethical Leadership Scale (ELS): Italian adaptation and exploration of the nomological network in a health care setting

“…The exploratory and confirmatory factorial analyses confirmed the mono‐dimensionality of the scale by reporting excellent saturations and model fit (Hair et al, ). As far as the correlations with other measures are concerned, the analyses allowed the aggregation of the measures surveyed at the centre level, with some caution in the case of the cynicism scale, for which the rwg value of 0.40 suggested only weak agreement among the employees (O'Neill, ). Then, the significant correlations with all the study variables in the expected direction confirmed the construct validity of the ELS in its factorial, convergent, discriminant and nomological facets.…”

Assessing the Quality of Mobile Phone Apps for Weight Management: User-Centered Study With Employees From a Lebanese University (Preprint)

“…Also, users might have had different interpretations of the items, as some IRA indices pointed to low or no agreement within 'engagement' items (MyFitnessPal and MyDietDiary), 'information' (MyDietDiary, MyPlate, and SparkPeople) and 'subjective quality' (Lark, MyFitnessPal, and MyDietDiary). While inter-rater agreement does not imply reliability [61,62], low agreement suggests a large degree of subjectivity in evaluating the apps. Furthermore, some users might have not understood some items, especially in the 'information' domain, which registered the highest non-response rates (22-39%).…”

“…For feasibility reasons (budget and time constraints), we could not ask all users to evaluate each app, hence allowing us to calculate inter-rater reliability. To overcome this limitation, we employed methodological adjustments (i.e., IRA estimates [59,61] and response-based weighted means [68]) to ensure the robustness of the responses obtained from the employees recruited in this study. Also, users evaluated the free version of the apps.…”

“…To ensure that ratings could be aggregated, we evaluated inter-rater agreement (IRA) following literature recommendations [61,62], using three families of indices: James et al's r WG(J) [63,64] (based on multiple null distributions) [65], Brown and Hauenstein's a WG(J) [66]; and the adjusted average deviation index A DMJ(adj) [67]. In addition to the arithmetic mean of each uMARS score, we calculated a "response databased weighted mean" (WDMEAN) [68].…”