An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Abstract

Background: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. Methods: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. Results: Redaction reduced, but did not eliminate, reviewers' ability to correctly guess features of identity. The primary, pre-registered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not pre-registered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen's d of 0.20 to 0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. Conclusions: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity. Funding: Funding was provided by the NIH.

Double‐blind peer review affects reviewer ratings and editor decisions at an ecology journal

“…Some find that papers authored by women are rated lower or have lower acceptance rates in academic journals (Fox et al, 2019; Murray et al, 2018; Walker et al, 2015; and references therein), but others have failed to find gender differences in peer review outcomes, or have even found that papers authored by women perform better than those authored by men (Lerback & Hanson, 2017; Squazzoni et al, 2021). Other biases, such as favouring papers by authors from the same country or that speak the same language as the reviewer (Murray et al, 2018), favouring authors from higher‐income (Demarest et al, 2014; Harris, Macinko, et al, 2017; Harris, Marti, et al, 2017; Kowal et al, 2022; Saposnik et al, 2014) or English‐speaking countries (Ross et al, 2006; Saposnik et al, 2014), discriminating based on author race (Nakamura et al, 2021), favouring papers by authors that are from prestigious institutions (Blank, 1991; Tomkins et al, 2017) or with prestigious reputations (Huber et al, 2022; Okike et al, 2016), favouring authors that are senior (Pleskac et al, 2021), or disfavouring newcomers to a discipline (Seeber & Bacchelli, 2017), can all distort the quality and fairness of peer review.…”

Peer reviewer’s dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

“…Similarly, redacting grant applicant identifiers may not eliminate reviewers' ability to second-guess applicants' characteristics, but there is evidence to suggest that it could reduce halo effects and improve fairness (Nakamura et al, 2021). Overall, therefore, it could be argued that a more effective and equitable peer-review system requires funders, applicants and peer reviewers themselves to collectively agree on what values should be prioritised -which may vary across grant schemes -and what courses of action better enshrine those values.…”

Reflections on race, ethnicity, and NIH research awards

“…Subsequent research has found limited evidence of bias in the review process ( Forscher, et al, 2019 ; Erosheva et al, 2020 ; Nakamura et al, 2021 ). Hoppe et al (2019) examined each stage of the NIH review process.…”