Double‐blind peer review affects reviewer ratings and editor decisions at an ecology journal

Abstract

1. There is substantial evidence that systemic biases influence the scholarly peer review process. Many scholars have advocated for double-blind peer review (also known as double-anonymous review) to reduce these biases. However, the effectiveness of double-blind peer review in eliminating biases is uncertain because few randomized trials have manipulated blinding of author identities for journal submissions and those that have are generally small or provide few insights on how it influences reviewer biases.2. In 2019, Functional Ecology began a large, randomized trial, using real manuscript submissions, to evaluate the various consequences of shifting to double-blind peer review. Research papers submitted to the journal were randomly assigned to be reviewed with author identities blinded to reviewers (double-blind review) or with authors identified to reviewers (single-blind review). In this paper, we explore the effect of blinding on the outcomes of peer review, examining reviewer ratings and editorial decisions, and ask whether author gender and/or location mediate the effects of review type.3. Double-blind review reduced the average success of manuscripts in peer review; | 1145Functional Ecology FOX et al. | INTRODUC TI ONThe assessment of research by experts in the subject, commonly called peer review, is the primary means by which granting agencies and journal editors assess the quality and importance of scholarly research. Yet there is a widespread perception among researchers that systemic biases influence the outcomes of peer review. In particular, there is concern that the gender, nationality, location or reputation of authors may influence how manuscripts are assessed by reviewers due to unconscious, or sometimes conscious, biases (Lee et al., 2013). Of all of the possible biases that can occur during peer review, bias against women has received the most attention, although studies are inconsistent in their outcomes. Some find that papers authored by women are rated lower or have lower acceptance rates in academic journals Murray et al., 2018;Walker et al., 2015; and references therein), but others have failed to find gender differences in peer review outcomes, or have even found that papers authored by women perform better than those authored by men (Lerback & Hanson, 2017;Squazzoni et al., 2021). Other biases, such as favouring papers by authors from the same country or that speak the same language as the reviewer (

Editors are biased too: An extension of Fox et al. (2023)'s analysis makes the case for triple‐blind review

“…Specifically, even an unbiased editor would still make a slightly biased decision due to the contribution of the biased reviewer scores to their decision, but establishing this exact null expectation is not possible without more information. We thus conclude that editor bias can only rigorously be assessed using the data from Fox et al (2023) in the double-blind treatment.…”

“…fit such models (Figure 1a) and found that there was a significant residual effect of the interaction between review treatment and gender but not either country HDI level or English dominance. In the absence of interpretation of these results by Fox et al (2023), we initially assumed-perhaps like other readers-that their intention was to test for editor bias, although subsequent correspondence with the authors revealed that they thought the model inadequate for such inference. Yet it is worth asking if this model could be a robust test for editor bias, for if so Fox et al (2023) have already provided important results relevant to the overlooked issue of editor bias.…”

“…Editors invite papers to be resubmitted based both on reviewer scores (blue path) and the editor's own appraisal of paper quality (orange path). (a) A model reported in Fox et al (2023), with a direct path from demography × review treatment to probability of resubmission, independent of reviewer score, may erroneously be interpreted as a test of editor bias. However, such an interpretation forgets that the review treatment is meaningless for editors, who are never blinded to author identities, and does not account for editors' independent appraisal of paper quality.…”

“…Lastly, we note that the onus is already on reviewers and editors to notify journals in the case of a conflict of interest, and this policy could be extended to cover cases where the reviewer and editor believe that they have guessed the identity of an anonymised author and have a conflict of interest (as implemented for the journal Ethics: Richardson, 2017). Fox et al (2023) report that more than half of reviewers correctly guessed the identities of authors in their double-blind study.…”

“…The logic here would be the effects of reviewer bias on the fates of papers are already completely accounted for by reviewer scores in the model, so a significant residual effect of demographic category by review treatment interaction must reflect a different source of bias-and the only bias remaining is that by editors. Indeed,Fox et al (2023)…”