Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study

Abstract

One of the main critiques of academic peer review is that inter-rater reliability (IRR) among reviewers is low. We examine an under-investigated factor possibly contributing to low IRR, reviewers’ diversity in their topic-criteria mapping (TC-mapping for short). It refers to differences among reviewers pertaining to which topics they choose to emphasize in their evaluations, and how they map those topics onto various evaluation criteria. In this paper we look at the review process of grant proposals in one funding agency to ask: how much do reviewers differ in TC-mapping, and do their differences contribute to low IRR? Through a content analysis of review forms submitted to a national funding agency (Science Foundation Ireland) and a survey of its reviewers, we find evidence of inter-reviewer differences in their TC-mapping. Using a simulation experiment we show that, under a wide range of conditions, even strong differences in TC-mapping only have a negligible impact on IRR. Although further empirical work is needed to corroborate simulation results, these tentatively suggest that reviewers’ heterogeneous TC-mappings might not be of concern for designers of peer review panels to safeguard inter-rater reliability. Peer Review https://publons.com/publon/10.1162/qss_a_00207