Reading Peer Review

Abstract

This Element describes for the first time the database of peer review reports at PLOS ONE, the largest scientific journal in the world, to which the authors had unique access. Specifically, this Element presents the background contexts and histories of peer review, the data-handling sensitivities of this type of research, the typical properties of reports in the journal to which the authors had access, a taxonomy of the reports, and their sentiment arcs. This unique work thereby yields a compelling and unprecedented set of insights into the evolving state of peer review in the twenty-first century, at a crucial political moment for the transformation of science. It also, though, presents a study in radicalism and the ways in which PLOS's vision for science can be said to have effected change in the ultra-conservative contemporary university. This title is also available as Open Access on Cambridge Core.

Does peer review improve the statistical content of manuscripts? A study on 27 467 submissions to four journals

“…The role of peer review in improving the quality of scientific publications has been subject to increasing scrutiny in recent research [ 28 ], which has led especially to more examination of current practices and standards [ 27 , 36 , 37 ]. However, this type of research only rarely integrates full data on manuscripts during each stage of the editorial process and data on review reports, at the same time covering different journals [ 38 , 39 ].…”

“…However, this type of research only rarely integrates full data on manuscripts during each stage of the editorial process and data on review reports, at the same time covering different journals [ 38 , 39 ]. Integrating data on manuscripts and reports is key to providing a context-specific picture of peer review and editorial processes, not to mention the possibility of assessing changes and revisions of manuscripts due to peer review [ 28 , 40 ]. Although difficult, pooling across-journal data is instrumental to examine the emergence of peer review practices that are shared in various communities [ 24 , 35 ].…”

“…To improve this type of research, removing obstacles against data sharing from publishers to the community and increasing interdisciplinary, multi-approach studies combining qualitative and quantitative research is needed [ 24 ]. Not only would this help us assess the developmental role of peer review more systematically, but also this type of research could inform guidelines and arrangements to improve the fairness of peer review [ 28 , 29 ] and improve our understanding of the multiple functions and dimensions of this complex social institution called peer review [ 4 , 50 ].…”

Innovating peer review, reconfiguring scholarly communication: An analytical overview of ongoing peer review innovation activities

“…While often criticized for its shortcomings, peer review continues to be widely seen as one among a range of distinct features that help set scientific knowledge apart from other forms of knowledge, thus warranting its special status in modern societies (Hackett and Chubin, 1990;Nicholas et al, 2015;Eve et al, 2021). Innovations in peer review processes -spurred especially by the pervasive digitization of many aspects of scholarly communication from the 1990s onward -have consequently drawn significant analytical attention from various disciplines.…”

The researchers using AI to analyse peer review

“…Another team worked with the publisher PLOS ONE and examined more than 2,000 reports from its database, looking at aspects including sentiment and tone 6 .…”