Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants

Abstract

Rationale We previously demonstrated absence of association between peer-review derived percentile ranking and raw citation impact in a large cohort of NHLBI cardiovascular R01 grants, but we did not consider pre-grant investigator publication productivity. We also did not normalize citation counts for scientific field, type of paper, and year of publication. Objective Determine whether measures of investigator prior productivity predict a grant’s subsequent scientific impact as measured by normalized citation metrics. Methods and Results We identified 1492 investigator-initiated de novo NHLBI R01 grant applications funded between 2001 and 2008 and linked the publications from these grants to their “InCites™” (Thompson Reuters) citation record. InCites™ provides a normalized citation count for each publication stratifying by year of publication, type of publication, and field of science. The co-primary endpoints for this analysis were the normalized citation impact per million dollars allocated and the number of publications per grant that have normalized citation rate in the top decile per million dollars allocated (“top-10% papers”). Prior productivity measures included the number of NHLBI-supported publications each principal investigator published in the 5 years before grant review and the corresponding prior normalized citation impact score. After accounting for potential confounders, there was no association between peer-review percentile ranking and bibliometric endpoints (all adjusted P > 0.5). However, prior productivity was predictive (P<0.0001). Conclusion Even after normalizing citation counts, we confirmed a lack of association between peer-review grant percentile ranking and grant citation impact. However, prior investigator publication productivity was predictive of grant-specific citation impact.

Competitive Science: Is Competition Ruining Science?

“…While it is often stated that more funding alone will not be adequate to fix science, more funding is an essential part of any effective solution. In this regard, a system that funds people instead of projects may be more rational given studies showing that this approach fosters higher-impact science (73) and that track record rather than project reviews is predictive of future researcher productivity (74,75). A greater emphasis should be placed on open-ended investigator-initiated research and less on targeted programs.…”

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

“…Lindner and Nakamura 21 found that the bibliometric indexes of publications from funded projects are not an appropriate measure of the scientific impact of an application. Similarly, Kaltman et al 22 and Doyle et al 23 found no association between grant percentile ranking and an applicant's bibliometric indexes. Derrick et al 24 evaluated whether citation metrics correlated with peer assessment of a researcher's influence on his/her field.…”

“…Furthermore, grant applications can be distorted by citation bias, for example, by choosing citations that support the proposed research, misrepresenting findings from cited research, and stating invested claims as fact. 28 Consistent with the above concerns, Kaltman et al 22 analyzed the outcomes of 1492 National Heart, Lung, and Blood Institute-funded R01s and found no association between percentile rank scores and subsequent citation impact of the project. They found, however, that the prior citation impact of the applicant was predictive of the citation impact of the grant project.…”

A manifesto for fair and equitable research funding in ecology

“…When the grant-awarding process has been systematically evaluated, there appears little evidence that peer-review scores reliably predict the subsequent success of research projects (Danthi et al 2014;Kaltman et al 2014;Danthi et al 2015). Although a study by Li and Agha (2015) indicated that panel rankings predicted good research outcomes (in terms of publication record), a subsequent reanalysis by Fang et al (2016) found no such effect.…”