NSF's ‘Big Pitch’ Tests Anonymized Grant Reviews

Bhattacharjee, Yudhijit

doi:10.1126/science.336.6084.969

Cited by 16 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The National Science Foundation recently anonymized a subset of research proposals and found that these proposals were evaluated very differently from those that retained the investigators’ names and affiliations. 20 Some researchers have suggested that the anonymized proposals allowed reviewers to focus more on the research question and the quality of the science than on the track record of the researcher. In addition, implicit bias and racial stereotypes may affect review outcomes.…”

Section: Discussionmentioning

confidence: 99%

Are Race, Ethnicity, and Medical School Affiliation Associated With NIH R01 Type 1 Award Probability for Physician Investigators?

et al. 2012

View full text Add to dashboard Cite

Purpose To analyze the relationship among NIH R01 Type 1 applicant degree, institution type, and race/ethnicity, and application award probability. Method The authors used 2000–2006 data from the NIH IMPAC II grants database and other sources to determine which individual and institutional characteristics of applicants may affect the probability of applications being awarded funding. They used descriptive statistics and probit models to estimate correlations between race/ethnicity, degree (MD or PhD), and institution type (medical school or other institution), and application award probability, controlling for a large set of observable characteristics. Results Applications from medical schools were significantly more likely than those from other institutions to receive funding, as were applications from MDs versus PhDs. Overall, applications from blacks and Asians were less likely than those from whites to be awarded funding; however, among applications from MDs at medical schools, there was no difference in funding probability between whites and Asians and the difference between blacks and whites decreased to 7.8 percentage points. The inclusion of human subjects significantly decreased the likelihood of receiving funding. Conclusions Compared with applications from whites, applications from blacks have a lower probability of being awarded R01 Type 1 funding, regardless of the investigator’s degree. However, funding probability is increased for applications with MD investigators and for those from medical schools. To some degree, these advantages combine so that applications from black MDs at medical schools have the smallest difference in funding probability compared with those from whites.

show abstract

Section: Discussionmentioning

confidence: 99%

Are Race, Ethnicity, and Medical School Affiliation Associated With NIH R01 Type 1 Award Probability for Physician Investigators?

et al. 2012

View full text Add to dashboard Cite

show abstract

“…Bias in the application review process may also play a role in funding rates by gender, race, or ethnicity. A National Science Foundation initiative to disentangle the effect of bias in review found that when proposals were anonymized, reviewers recommended a different set of proposals to be funded (22). A more recent study found that reviewers assigned significantly worse evaluation scores to female principal investigators (PIs) than male PIs, which suggests that subconscious gender bias may lead reviewers to implicitly hold female applicants to higher standards than male applicants (23).…”

mentioning

confidence: 99%

Examining trends in the diversity of the U.S. National Institutes of Health participating and funded workforce

et al. 2018

View full text Add to dashboard Cite

Here, we use recent U.S. National Institutes of Health (NIH) data to document trends in the NIH-funded workforce over time. Consistent with previous studies that were initiated by NIH, we find that the number of scientists funded on competing R01-equivalent (R01 Eq.) and research project grants (RPGs) increased 2-5% per year between 2009 and 2016. Primary beneficiaries of this growth were experienced investigators (Exp), whereas the share of funding awarded to early-stage investigators (ESIs) and new investigators (NIs) declined. The decline occurred even after NIH instituted the New and Early-Stage Investigator policy in 2009. When we evaluate the investigator pool, we find that women and racial and ethnic minorities represent a higher percentage of NIs and ESIs relative to Exp. Thus, trends of diminishing support for NIs and ESIs may negatively impact the diversity of the current and future biomedical research workforce. We find some recent gains among women and Hispanics as part of the applicant and awardee pool for both R01 Eq. and RPGs, but significant, large gaps persist among nationally underrepresented racial minorities. Our findings suggest a need to prioritize investments and support of ESIs and NIs, groups in which women and racial and ethnic minorities represent a larger proportion of the applicant pool, to enhance diversity in the NIH-funded workforce.-Nikaj, S., Roychowdhury, D., Lund, P. K., Matthews, M., Pearson, K. Examining trends in the diversity of the U.S. National Institutes of Health participating and funded workforce.

show abstract

“…A large-scale study of applications to the National Science Foundation of Korea found that applications reviewed by previous or current affiliates were more likely to be successful (Jang et al 2016). This is further supported by research at the US National Science Foundation, described by Bhattacharjee (2012), where several different comparisons of panel assessments of traditional full proposals and shorter anonymised versions of the proposals showed weak correlations. Luukkonen (2012) notes that the effectiveness of panel debate, which is expected to counter crude forms of cronyism, can be limited since often panels must cover a wide area of research, meaning each specific area is represented by only a few experts.…”

Section: Is Peer Review Fair?mentioning

confidence: 78%

“…While some funding bodies now routinely attempt to anonymise proposals before passing them on to reviewers, there is some dispute as to whether anonymisation is truly possible. Some authors contend that some degree of identification is always possible from anonymised research proposals (Bhattacharjee 2012) Improving fairness and removing bias using a lottery system…”

Section: Improving Fairness Through Blindingmentioning

confidence: 99%

What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies

Guthrie¹,

Ghiga²,

Wooding³

2018

View full text Add to dashboard Cite

An updated review of the literature and six case studies RAND Europe is a not-for-profit organisation whose mission is to help improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors. Limited Print and Electronic Distribution RightsThis document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions. Support RANDMake a tax-deductible charitable contribution at www.rand.org/giving/contribute www.rand.org www.rand.org/randeurope For more information on this publication, visit www.rand.org/t/RR1822Published by the RAND Corporation, Santa Monica, Calif., and Cambridge, UK © Copyright 2018 RAND CorporationR® is a registered trademark.ii PrefaceIn 2009, RAND Europe conducted a literature review in order to assess the effectiveness and efficiency of peer review for grant funding. This report presents an update to that review to reflect new literature on the topic, and adds case studies exploring peer review practice at six international funders.This report was produced with funding from the Canadian Institutes of Health Research. It will be of interest to government officials dealing with research funding policy, research funders including governmental and charitable funders, research institutions, researchers, and research users. Although the case studies focus on biomedical and health research, the literature review takes a broader scope and it is likely the findings may be of relevance to wider research fields.For further information on this publication please contact: SummaryThough often viewed as the 'gold standard' process of quality assurance for research, grant peer review has also received significant criticism from both within and outside academia. Detractors highlight inefficiency and structural flaws that compromise its effectiveness in allocating funding. In 2009 we conducted a review of the literature to evaluate these criticisms. 1 This new report updates that literature review, and also provides case studies of current practice across six major international biomedical and health research funders. The work was commissioned by the Canadian Institutes of Health Research (CIHR) with the aim of supporting the ongoing review of their peer review system, particularly the forthcoming work of the Peer Review Expert Panel convened to review the design and adjudication processes of CIHR's investigator-initiated research programmes. What is grant peer review?In its most basic form, peer review involves academic reviewers in the process...

show abstract

NSF's ‘Big Pitch’ Tests Anonymized Grant Reviews

Abstract: An experiment being conducted at the U.S. National Science Foundation indicates that NSF may be turning down deserving research proposals because of potential biases in the grant-review process.

Cited by 16 publications

References 0 publications

Are Race, Ethnicity, and Medical School Affiliation Associated With NIH R01 Type 1 Award Probability for Physician Investigators?

Are Race, Ethnicity, and Medical School Affiliation Associated With NIH R01 Type 1 Award Probability for Physician Investigators?

Examining trends in the diversity of the U.S. National Institutes of Health participating and funded workforce

What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies

Contact Info

Product

Resources

About