The argument for diversifying the NIH grant portfolio

Peifer, Mark

doi:10.1091/mbc.e17-07-0462

Cited by 17 publications

(24 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, as the decisions become more subjective, the likelihood for bias increases, and thus much effort must be focused on ensuring the fidelity and equity of the review process. It is likely unavoidable that some meritorious research will not be funded, putting more pressure on research funding administrators to incorporate into the final funding decisions considerations of portfolio diversification, programmatic concerns, promotion of collaborations and risk considerations (Galis et al, 2012;Janssens et al, 2017;Peifer, 2017;Wahls, 2018). These considerations, as well as the creation of new funding mechanisms (e.g., funds for early career investigators; Kaiser, 2017) should complement research into peer review processes.…”

Section: Discussionmentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

Peer review is used commonly across science as a tool to evaluate the merit and potential impact of research projects and make funding recommendations. However, potential impact is likely to be difficult to assess ex-ante; some attempts have been made to assess the predictive accuracy of these review decisions using impact measures of the results of the completed projects. Although many outputs, and thus potential measures of impact, exist for research projects, the overwhelming majority of evaluation of research output is focused on bibliometrics. We review the multiple types of potential impact measures with an interest in their application to validate review decisions. A review of the current literature on validating peer review decisions with research output impact measures is presented here; only 48 studies were identified, about half of which were US based and sample size per study varied greatly. 69% of the studies employed bibliometrics as a research output. While 52% of the studies employed alternative measures (like patents and technology licensing, post-project peer review, international collaboration, future funding success, securing tenure track positions, and career satisfaction), only 25% of all projects used more than one measure of research output. Overall, 91% of studies with unfunded controls and 71% of studies without such controls provided evidence for at least some level of predictive validity of review decisions. However, several studies reported observing sizable type I and II errors as well. Moreover, many of the observed effects were small and several studies suggest a coarse power to discriminate poor proposals from better ones, but not amongst the top tier proposals or applicants (although discriminatory ability depended on the impact metric). This is of particular concern in an era of low funding success, where many top tier proposals are unfunded. More research is needed, particularly in integrating multiple types of impact indicators in these validity tests, as well as considering the context of the research outputs relative to goals of the research program and concerns for reproducibility, translatability and publication bias. In parallel, more research is needed focusing on the internal validity of review decision making procedures and reviewer bias.Keywords: research funding, peer review, impact metrics, validity, grant GRANT PEER REVIEW AND IMPACT ASSESSMENTMost would generally agree the purpose of biomedical research is to advance knowledge for societal benefit, with the hope of favorably impacting disease outcomes and improving global health. Indeed, the National Institutes of Health (NIH ), the world's largest funder of biomedical research, characterizes their mission as to "seek fundamental knowledge about the nature and Gallo and GlissonReview Validity via Impact Measures behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability" (NIH, 2017). To help select which research projects to fund to achieve...

show abstract

Section: Discussionmentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

show abstract

“…When a principal investigator's grant funding amount exceeds his or her capacity to carry out grant-related duties, such as the day-to-day management of laboratory personnel, productivity falls (19,20). Such diminishing marginal returns apply at the population scale for dollars per award and dollars per investigator (21)(22)(23)(24).…”

Section: Diminishing Marginal Returnsmentioning

confidence: 99%

The National Institutes of Health needs to better balance funding distributions among US institutions

Wahls

2019

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

The NIH is the federal steward of biomedical research in the United States. Taxpayers fund the NIH; the NIH supports research into the underlying biology, etiology, and treatment of diseases; and benefits of that research are returned to taxpayers. This is a large and complex enterprise, but at its core are two fundamental principles. The NIH is obligated to distribute its research grants and grant dollars in a fair and impartial manner among qualified investigators nationwide and to maximize the return on taxpayers' investments. But to meet these obligations, the NIH must establish a better-balanced distribution of funding among institutions. Here, I describe why such changes are needed and propose a specific mechanism for how they could be implemented. Diversity Matters The director of the NIH, Francis Collins, has recognized that "the US biomedical research workforce does not currently mirror the nation's population demographically" and that "recruiting and retaining a The NIH should rethink how it doles out funding. Studies indicate that research output does not grow linearly with the amount of research project grant support-and that the greatest rates of return are achieved with intermediate levels of funding. Image credit: Dave Cutler (artist).

show abstract

“…Giving the lion’s share of grant dollars to a small minority of institutions seems counterproductive and wasteful—whether or not the disparities in funding are driven by bias. As is the case for the distribution of research dollars among individual investigators (Lorsch, 2015; Mongeon et al, 2016; Peifer, 2017a, 2017b; Wahls, 2017, 2018), a more egalitarian distribution of funding among institutions would yield greater collective gains for the research enterprise and the taxpayers who support it.…”

Section: Discussionmentioning

confidence: 99%

“…Diversity in scientific research includes the perspectives and creative ideas that are harnessed, the model systems and experimental tools employed, the types of investigators supported, and the regions in which research is conducted. Multiple levels of diversity increase the likelihood of scientific breakthroughs and maximize the return on taxpayers’ investments in federally sponsored research (Lorsch, 2015; Peifer, 2017a). Unfortunately, there are barriers to maximizing diversity.…”

Section: Introductionmentioning

confidence: 99%

“…This creates a barrier for the entry of talented young scientists into the biomedical workforce, threatening the future of the research enterprise (Carr, 2013). Similarly, the approximately 70% of awardees who hold a single NIH grant are at increased risk of losing that support, their research laboratories, and even their livelihood (Peifer, 2017a). Consequently, scientists, agency officials and organizations such as the Federation of American Societies for Experimental Biology have advocated for a more equitable distribution of funding among investigators to help sustain the biomedical research enterprise (e.g., Alberts et al, 2014; FASEB, 2015; Lorsch, 2015; Peifer, 2017a; Wahls, 2018).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

High cost of bias: Diminishing marginal returns on NIH grant funding to institutions

Wahls

2018

Preprint

View full text Add to dashboard Cite

16Scientific output is not a linear function of amounts of federal grant support to individual 17 investigators. As funding per investigator increases beyond a certain point, productivity 18 decreases. This study reports that such diminishing marginal returns also apply for National 19Institutes of Health (NIH) research project grant funding to institutions. Analyses of data (2006-20 2015) for a representative cross-section of institutions, whose amounts of funding ranged from 21 $3 million to $440 million per year, revealed robust inverse correlations between funding (per 22 institution, per award, per investigator) and scientific output (publication productivity and citation 23 impact productivity). Interestingly, prestigious institutions had on average 65% higher grant 24 application success rates and 50% larger award sizes, whereas less-prestigious institutions 25 produced 65% more publications and had a 35% higher citation impact per dollar of funding. 26These findings suggest that implicit biases and social prestige mechanisms (e.g., the Matthew 27 effect) have a powerful impact on where NIH grant dollars go and the net return on taxpayers' 28 investments. They support evidence-based changes in funding policy geared towards a more 29 equitable, more diverse and more productive distribution of federal support for scientific 30 research. Success rate/productivity metrics developed for this study provide an impartial, 31 empirically based mechanism to do so. 32 33 Keywords 34 35 Science policy; peer review; bias; implicit bias; social prestige mechanisms; Matthew effect 36 37Call-Out Quotes 38 39 "Giving the lion's share of grant dollars to a small minority of institutions seems 40 counterproductive and wasteful-whether or not the disparities in funding are driven by bias." 41 42 "A more egalitarian distribution of funding among institutions would yield greater collective gains 43 for the research enterprise and the taxpayers who support it." 48 types of investigators supported, and the regions in which research is conducted. Multiple 49 levels of diversity increase the likelihood of scientific breakthroughs and maximize the return on 50 taxpayers' investments in federally sponsored research (Lorsch, 2015; Peifer, 2017a). 51Unfortunately, there are barriers to maximizing diversity. 53A landmark study in Science reported that black investigators are much less likely to get their 54 National Institutes of Health (NIH) research grant applications funded than white applicants, 55 even after for controlling for other factors (Ginther et al., 2011). There are also large differences 56 in success rates for investigators grouped by age (Levitt & Levitt, 2017). While there does not 57 seem to be a gender gap for new NIH grants, female applicants have lower success rates than 58 their male counterparts for competitive renewals (Kaatz et al., 2016; Magua et al., 2017; 59 Pohlhaus et al., 2011). There are also large differences in success rates for investigators 60 grouped by state (Wahls, 2016). The differences in suc...

show abstract

The argument for diversifying the NIH grant portfolio

Cited by 17 publications

References 17 publications

External Tests of Peer Review Validity Via Impact Measures

External Tests of Peer Review Validity Via Impact Measures

The National Institutes of Health needs to better balance funding distributions among US institutions

High cost of bias: Diminishing marginal returns on NIH grant funding to institutions

Contact Info

Product

Resources

About