Inter-Rater Agreement, Data Reliability, and The Crisis of Confidence in Psychological Research

Button, Cathryn M.; Snook, Brent; Grant, Malcolm J.

doi:10.20982/tqmp.16.5.p467

Cited by 5 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A rubric must be reliable so that different assessors can use the rubric consistently (Reynders et al, 2020). Interrater agreement is the most common indicator of good research when two or more raters are involved to provide an assessment of an instrument so that the instrument's reliability can be assessed (Button et al, 2020;Tong et al, 2020).…”

Section: Figure 3 Examples Of Student Answers On the Supposition And ...mentioning

confidence: 99%

Development of Critical Thinking Skills Instruments : Cases for Essay Tests

Hidayati,

Suryanti,

Rahmayumita

et al. 2024

j. kependidikan. has. penelit. kaji. kepustakaan. bid. pendidik

View full text Add to dashboard Cite

This research aims to develop a rubric used to assess students’ critical thinking. This rubric refers to aspects of Ennis' critical thinking. The research method begins with determining the scope of the rubric so that an initial version of the rubric can be prepared. An initial version of the rubric received peer review. Next, the rubric was tested to assess critical thinking tests with essay-type questions. Data analysis was carried out using Cohen's kappa, while aspects of the critical thinking rubric were analyzed descriptively. The research results show that each aspect of critical thinking has been equipped with descriptors that can be used by raters. The level of reliability based on the kappa value of agreement is quite good, which means that different raters can use the rubric consistently. The value of agreement in the basic clarification aspect is 0.33 (fair), bases for a decision (0.571 in the moderate category), inference (0.455, moderate category), advanced clarification (0.250, fair category, supposition and integration (0.250, fair category), and strategies and tactics (0.182, poor category). Then, it can be concluded that the critical thinking rubric developed can be used to assess critical thinking abilities. As a recommendation, of course, this rubric cannot fulfill all types of critical thinking questions tested. It is also necessary to develop rubrics for question types other than essays to accommodate the wider need for critical thinking rubrics.

show abstract

Section: Figure 3 Examples Of Student Answers On the Supposition And ...mentioning

confidence: 99%

Development of Critical Thinking Skills Instruments : Cases for Essay Tests

Hidayati,

Suryanti,

Rahmayumita

et al. 2024

j. kependidikan. has. penelit. kaji. kepustakaan. bid. pendidik

View full text Add to dashboard Cite

show abstract

“…Pri percentuálnom súhlase to ale nemusí byť nakoniec taký veľký faux pas keď ho použijeme, ako sme už spomínali vyššie. Jeho popularitu je možné pripísať viacerým faktorom (Button et al, 2020). Za prvé, rady odborníkov sú zmätočné (ak medzi samými výskumníkmi -metodológmi panuje nezhoda v tom, aký index je najlepší, aj praktický výskumníci majú v tom zmätok).…”

Section: Porovnanie Percentuálneho Súhlasu Koeficientu κ a Koeficient...unclassified

Options, Problems and Guidelines for Measuring Interrater Agreement - a Descriptive Approach

Kočišová

2024

Czech Sociological Review

View full text Add to dashboard Cite

show abstract

“…Gwet's AC 1 makes a c a function of both category and skew. Although many other indices are available and new indices continue to emerge, only these seven are in regular use and continue to be recommended or advocated, according to comprehensive reviews [14,[20][21][22][23][24][25][26].…”

mentioning

confidence: 99%

Interrater reliability estimators tested against true interrater reliabilities

Zhao

Feng

et al. 2022

BMC Med Res Methodol

View full text Add to dashboard Cite

Background Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters’ random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies. Methods We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested. Results The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott’s π, Cohen’s κ and Krippendorff’s α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet’s AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al’s S ranked behind AC1, and Perreault and Leigh’s Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating. Conclusion The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

show abstract

Inter-Rater Agreement, Data Reliability, and The Crisis of Confidence in Psychological Research

Cited by 5 publications

References 26 publications

Development of Critical Thinking Skills Instruments : Cases for Essay Tests

Development of Critical Thinking Skills Instruments : Cases for Essay Tests

Options, Problems and Guidelines for Measuring Interrater Agreement - a Descriptive Approach

Interrater reliability estimators tested against true interrater reliabilities

Contact Info

Product

Resources

About