Predicting Productivity Returns on Investment

Abstract

There are conflicting data regarding the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute between 1980 and 2011. Our outcomes focus on “Top-10%” papers, meaning papers that were cited more often than 90% of other papers on the same topic, of the same type (e.g. article, editorial), and published in the same year. The 6873 grants yielded 62,468 papers, of which 13,507 (or 22%) were Top-10% papers. There was a modest association between better grant percentile ranking and number of top-10% papers. However, discrimination was poor (area under ROC 0.52, 95% CI 0.51–0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% papers per $million spent. Hence, the seemingly conflicting findings regarding peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best NIH should use peer review assessments to make complex funding decisions.

The argument for diversifying the NIH grant portfolio

“…There has been much discussion of the statistical approaches used and what the curve really looks like at the far-right end, among the best-funded laboratories. With regard to this point, the NIH data fit well with that from several other published studies, all of which documented diminishing returns as funding levels increased ( Fortin and Currie, 2013 ; Xie, 2014 ; Conti and Liu, 2015 ; Cook et al , 2015 ; Doyle et al , 2015 ; Lauer et al , 2015 ; Lorsch, 2015 ; Mongeon et al , 2016 ). Third, as noted above, it is impossible to predict from where the most important discoveries will emerge.…”

What do we know about grant peer review in the health sciences?

“…Studies comparing percentile application rankings with the research’s subsequent bibliometric performance found no association ( Danthi et al , 2014 ; Danthi et al , 2015 ; Doyle et al , 2015 ; Fang et al , 2016 ; Kaltman et al , 2014 ; van den Besselaar & Sandström, 2015 ). Two further such studies found that grant review outcomes only weakly predict bibliometric performance ( Lauer et al , 2015 ; Reinhart, 2009 ). Bibliometric analyses are by no means perfect measures of performance – only capturing a proxy of academic performance ( Belter, 2015 ).…”

Biases in grant proposal success rates, funding rates and award sizes affect the geographical distribution of funding for biomedical research

“…Congruent findings have been made for research supported by other NIH institutes and federal agencies. Further analyses of NHLBI award data ( Kaltman et al, 2014 ; Lauer et al, 2015 ), and of data from the National Institute of General Medical Sciences (NIGMS) ( Berg, 2012 ), the National Institute of Mental Health (NIMH) ( Doyle et al, 2015 ), and the NSF ( Scheiner & Bouchie, 2013 ) have each revealed little or no association between grant proposal scores and scientific outcomes of funded proposals. To the extent tested the findings apply for multiple measures of outcome (e.g., publication rate, number of highly cited publications, overall citation impact), even after accounting for additional variables.…”

“…To the extent tested the findings apply for multiple measures of outcome (e.g., publication rate, number of highly cited publications, overall citation impact), even after accounting for additional variables. (A cogent description of parameters affecting the interpretation of bibliometric data as indicators of scientific productivity, along with analogies understandable by a lay audience, can be found in Lauer et al (2015) . For example, a modest association between grant percentile score and scientific outcome, as assessed by highly cited publications per grant, disappears when adjusted for award size.)…”