Emerging trends in peer review—a survey

Abstract

“Classical peer review” has been subject to intense criticism for slowing down the publication process, bias against specific categories of paper and author, unreliability, inability to detect errors and fraud, unethical practices, and the lack of recognition for unpaid reviewers. This paper surveys innovative forms of peer review that attempt to address these issues. Based on an initial literature review, we construct a sample of 82 channels of scientific communication covering all forms of review identified by the survey, and analyze the review mechanisms used by each channel. We identify two major trends: the rapidly expanding role of preprint servers (e.g., ArXiv) that dispense with traditional peer review altogether, and the growth of “non-selective review,” focusing on papers' scientific quality rather than their perceived importance and novelty. Other potentially important developments include forms of “open review,” which remove reviewer anonymity, and interactive review, as well as new mechanisms for post-publication review and out-of-channel reader commentary, especially critical commentary targeting high profile papers. One of the strongest findings of the survey is the persistence of major differences between the peer review processes used by different disciplines. None of these differences is likely to disappear in the foreseeable future. The most likely scenario for the coming years is thus continued diversification, in which different review mechanisms serve different author, reader, and publisher needs. Relatively little is known about the impact of these innovations on the problems they address. These are important questions for future quantitative research.

What is open peer review? A systematic review

“…Traditional peer review operates as either “single-blind”, where authors do not know reviewers’ identities, or “double-blind”, where both authors and reviewers remain anonymous. Double-blind reviewing is more common in the Arts, Humanities and Social Sciences than it is in STEM (science, technology, engineering and medicine) subjects, but in all areas single-blind review is by far the most common model ( Walker & Rocha da Silva, 2015). A main reason for maintaining author anonymity is that it is assumed to tackle possible publication biases against authors with traditionally feminine names, from less prestigious institutions or non-English speaking regions ( Budden et al , 2008; Ross et al , 2006).…”

“…Based on anecdotal evidence, Walker & Rocha da Silva (2015) advise that “[r]eports from participants are generally but not universally positive”. To the knowledge of the author, the only experimental study that has specifically examined interaction among reviewers or between reviewers and authors is that of Jeffrey Leek and his colleagues, who performed a laboratory study of open and closed peer review based on an online game and found that “improved cooperation does in fact lead to improved reviewing accuracy.…”

“…In response to these criticisms, a wide variety of changes to peer review have been suggested (see the extensive overview in Walker & Rocha da Silva, 2015). Amongst these innovations, many have been labelled as “open peer review” at one time or another.…”

A multi-disciplinary perspective on emergent and future innovations in peer review

“…While peer review is still generally perceived as key to quality control for research, it has been argued that mistakes are becoming more frequent in the process ( Margalida & Colomer, 2016; Smith, 2006), and that peer review is not being applied as rigorously as generally perceived. As a result, it has become the target of widespread criticism, with a range of empirical studies investigating the reliability, credibility and fairness of the scholarly publishing and peer review process (e.g., ( Bruce et al , 2016; Cole, 2000; Eckberg, 1991; Ghosh et al , 2012; Jefferson et al , 2002; Kostoff, 1995; Ross-Hellauer, 2017; Schroter et al , 2006; Walker & Rocha da Silva, 2015)). In response to this, initiatives like the EQUATOR network ( equator-network.org) have been important to improve the reporting of research and its peer review according to standardised criteria.…”

Contract reform: Who's to blame?

“…2 Post-publication peerreview has been suggested as an alternative but it is hampered by inadequate participation of the scientific community, 3 although there are portals that have been successful in using such platforms. 4 However, the new models have not dented the traditional system of publishing. In 2013, there were 1.8 million peer-reviewed articles published at a rate of one article every 18 seconds leading to an enormous strain on peer-review.…”