A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants

Abstract

BackgroundThis paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree.Methodology/Principal FindingsAltogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen's Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed.Conclusions/SignificanceStudies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).

Reliability of reviewers' ratings when using public peer review: a case study

“…Building on these statements, the finding of this study -low (k) or reasonable (ICC) inter-rater reliability -should be interpreted not as an indication that the quality of the ACP peer review process is low but instead as a general characteristic of journal manuscript reviewing 10 that possibly has a positive effect on the predictive validity of manuscript selection decisions. In recent years, some ways to increase inter-rater reliability in journal peer review have been suggested, such as including a greater number of reviewers.…”

Psychoinformatics

“…replication is under-emphasized, analysis and reporting standards can be lax (Simmons, Nelson, & Simonsohn, 2011), and conventional peer review is unreliable (Bornmann, Mutz, & Daniel, 2010). Thereʼs a tremendous incentive to develop tools and platforms that can help address such problems.…”

Dimensions of trust in scholarly communication: Problematizing peer review in the aftermath of John Bohannon's “Sting” in science

“…Since it is essentially the same researchers who publish and review, this further increases the workload. Second, the peer-review process per se has been increasingly debated, and we find a growing amount of studies showing problems related to the process, including bias and nepotism, as well as problems with interrater reliability, in the peer-based assessment of articles, research proposals, and evaluations of research institutions (e.g., Bornmann, 2011;Bornmann et al, 2013;Garcia et al, 2015;Lee et al, 2013;Wennerås & Wold, 1997). Not only open access, but also subscription-based journals have been forced to retract large numbers of articles (Steen et al, 2013; To trace the reactions and comments on Bohannon's article, the Open Access Tracking Project Primary (OATP Primary) was used to identify web documents, which reported and commented on Bohannon's article.…”