‘Conditional Acceptance’ (additional experiments required): A scoping review of recent evidence on key aspects of Open Peer Review

Abstract

Diverse efforts are underway to reform the journal peer review system. Combined with growing interest in Open Science practices, Open Peer Review (OPR) has become of central concern to the scholarly community. However, what Open Peer Review is understood to encompass and how effective some of its elements are in meeting the expectations of the peer review system, are uncertain. This scoping review updates previous efforts to summarise research on OPR to date. Following the PRISMA methodological framework, it addresses the question: “What evidence has been reported in the scientific literature from 2017 to date regarding uptake, attitudes, and efficacy of two key aspects of Open Peer Review (Open Identities and Open Reports)?” The review identifies, analyses and synthesises 52 studies matching inclusion criteria, finding that OPR is growing, but still far from common practice. Our findings indicate positive attitudes towards Open Reports and more sceptical approaches to Open Identities. Changes in reviewer behaviour seem limited. and no evidence for lower acceptance rates of review invitations or slower turnaround times is reported. Concerns about power dynamics and potential backfiring on critical reviews are in need of further experimentation. We conclude that elements of OPR seem to be gaining acceptance, but more experimentation is needed. Evidence still mainly consists of either survey data or case studies of individual or few journals, not allowing for generalisability across fields and journals, and revealing no studies which compare the quality of review under Open Identities or Open Reports versus other modes of peer review.

Where are the carrots? A proposal to start crediting peer reviewers for their contribution to science

“…We are aware that some academics are sceptical of open identities (Ross-Hellauer & Horbach, 2022), and that the fear of retribution can deter potential reviewers, but we believe that the positive effects brought on by the increased incentives far outweigh the possible deterring effects of open identities. If a reviewer wants to be anonymous for some reason, that same reviewer could opt to do so, but will in turn not receive public recognition for the review.…”

“…Furthermore, the review reports have additional potential as a learning tool for students and researchers and could serve as a source of inspiration and guidance for others faced with similar challenges (i.e., using similar methods, design, or theoretical framework; Schmidt et al, 2018). We are aware that some academics are sceptical of open identities (Ross‐Hellauer & Horbach, 2022), and that the fear of retribution can deter potential reviewers, but we believe that the positive effects brought on by the increased incentives far outweigh the possible deterring effects of open identities. If a reviewer wants to be anonymous for some reason, that same reviewer could opt to do so, but will in turn not receive public recognition for the review.…”

Open peer review urgently requires evidence: A call to action

“…Despite these high expectations, the evidence-base on which such claims rest is very thin [ 6 , 10 ]. In particular, there is still minimal information available on the fundamental question of the extent to which OPR affects the quality and trustworthiness of the items under review.…”

“…The initial step should be to extend and validate the agenda itself through cocreative methods. Although our prioritization of questions and methods draws heavily from a recent preprint that surveys the literature [ 10 ], we believe that to maximize uptake, the plan of action should be community driven.…”