The limitations to our understanding of peer review

Abstract

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.

Time to stop the exploitation of free academic labour

“…This includes, at a bare minimum, financial reimbursement, authoritative standards and checklists, and transparency and objectivity. There is absolutely no way that peer review can maintain its standing as a professional form of quality control without these things (14).…”

How open science is fighting against private, proprietary publishing platforms

“…Peer review, the supposed "golden standard" of scholarly research, remains curiously poorly understood both empirically and theoretically (24), with publishers again mostly unwilling to release this vital information for further study (25). Critical data and code needed to verify research results are often hidden or locked away silos, un-usable by the majority of people, and often we waste countless hours trying to reproduce the version of research contained in PDFs rather than reproduce actual research (26).…”

Innovating editorial practices: academic publishers at work

“…Implementation of novel review procedures seems to be restricted to specific niches (specialties, publishing platforms), with the exception of the implementation of text similarity software or 'plagiarism scanners' [8]. Slow adoption may be partly explained by a lack of systematic evidence of their effectiveness [12]. Nevertheless, given the fierce promotion by their advocates [13], it may seem strange that new review practices do not convince a wider set of journals.…”