What is open peer review? A systematic review

Abstract

Background: “Open peer review” (OPR), despite being a major pillar of Open Science, has neither a standardized definition nor an agreed schema of its features and implementations. The literature reflects this, with a myriad of overlapping and often contradictory definitions. While the term is used by some to refer to peer review where the identities of both author and reviewer are disclosed to each other, for others it signifies systems where reviewer reports are published alongside articles. For others it signifies both of these conditions, and for yet others it describes systems where not only “invited experts” are able to comment. For still others, it includes a variety of combinations of these and other novel methods. Methods: Recognising the absence of a consensus view on what open peer review is, this article undertakes a systematic review of definitions of “open peer review” or “open review”, to create a corpus of 122 definitions. These definitions are then systematically analysed to build a coherent typology of the many different innovations in peer review signified by the term, and hence provide the precise technical definition currently lacking. Results: This quantifiable data yields rich information on the range and extent of differing definitions over time and by broad subject area. Quantifying definitions in this way allows us to accurately portray exactly how ambiguously the phrase “open peer review” has been used thus far, for the literature offers a total of 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature. Conclusions: Based on this work, I propose a pragmatic definition of open peer review as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the ethos of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process.

A multi-disciplinary perspective on emergent and future innovations in peer review

“…While peer review is still generally perceived as key to quality control for research, it has been argued that mistakes are becoming more frequent in the process ( Margalida & Colomer, 2016; Smith, 2006), and that peer review is not being applied as rigorously as generally perceived. As a result, it has become the target of widespread criticism, with a range of empirical studies investigating the reliability, credibility and fairness of the scholarly publishing and peer review process (e.g., ( Bruce et al , 2016; Cole, 2000; Eckberg, 1991; Ghosh et al , 2012; Jefferson et al , 2002; Kostoff, 1995; Ross-Hellauer, 2017; Schroter et al , 2006; Walker & Rocha da Silva, 2015)). In response to this, initiatives like the EQUATOR network ( equator-network.org) have been important to improve the reporting of research and its peer review according to standardised criteria.…”

“…For a detailed comparison of double versus single blind review, Snodgrass (2007) provides an excellent summary. The advent of “open peer review” introduced substantial additional complexity into the discussion ( Ross-Hellauer, 2017). …”

“…Much of this can be ascribed to tensions between evolving cultural practices, social norms, and the different stakeholder groups engaged with scholarly publishing. This reluctance is emphasized in recent surveys, for instance the one by Ross-Hellauer (2017) suggests that while attitudes towards the principles of OPR are rapidly becoming more positive, faith in its execution is not. We can perhaps expect this divergence due to the rapid pace of innovation, which has not led to rigorous or longitudinal evidence that these models are superior to the traditional process at either a population or system-wide level.…”

“…It comes in various flavors that result from different approaches to the relative timing of the review in the publication cycle, the reciprocal transparency of the process, and the contrasting and disciplinary practices ( Ross-Hellauer, 2017). Such interdisciplinary differences have made the study and understanding of peer review highly complex, and implementing any systemic changes to peer review is fraught with the challenges of synchronous adoption between heterogeneous communities often with vastly different social norms and practices.…”

Can editors save peer review from peer reviewers?

“…As a result, the fairness, reliability, transparency, and sustainability of peer review have been repeatedly questioned [4–9], with emphasis on a slew of potential biases by all parties involved [10]. Equally long as the litany of debated issues is the list of proposed solutions, including alternatives to traditional peer review such as preprint repositories [11], double-blind peer review [12], and open peer review [13–15]. Many of those ideas have in turn not been definitively shown to outperform peer review [16], and are met with skepticism from proponents of classical peer review [17, 18].…”