Towards Theorizing Peer Review

Abstract

Academic peer review is seriously undertheorized because peer review studies focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe the main characteristics of the peer review literature, and, based on this, I argue why theory is useful in research on peer review. I then present some theoretical efforts on peer review and conclude by encouraging peer review researchers to be more theoretically engaged. This invitation to theory building complements recent roadmaps and calls that have emphasized that we need better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.

FENS‐Kavli Network of Excellence: Postponed, non‐competitive peer review for research funding

“…Justification of publication peer review is thus based on its ability to ensure an absolute minimum of research quality; in contrast, justification of grant proposal peer review is based on its ability to produce a reliable ranking of relative research quality. Low inter-rater reliability as a 'hallmark of peer review' (Hug, 2021) is thus relatively unproblematic in case of publication peer review as long as one of the reviewers catches major flaws of a study. In contrast, a failure to reliably rank research proposals questions the very core of the rationale to use a procedure as costly and inefficient as competitive peer review for the distribution of research funding.…”