The Burroughs Wellcome Career Award in the Biomedical Sciences

Abstract

The Career Award in Biomedical Sciences (CABS) was launched by the Burroughs Wellcome Fund to assist talented scientists to cross the "bridge" that connects postdoctoral training with independent investigator status. This study examines the causal effects of CABS, using both conventional methods and a newer method (propensity score analysis) in attempting to account for selection biases (e.g., selecting the "best and the brightest") in estimating the award's impact. The results indicated that grantees outperformed unsuccessful applicants in obtaining a faculty position, competing for an R01 grant, and publishing in prestigious journals. However, it was nearly impossible to remove all the effects of selection bias from estimates of program effects. Because selection bias is a pervasive problem in assessing these programs, recommendations for improving evaluations of similar merit-based programs are offered.

External Tests of Peer Review Validity Via Impact Measures

“…Multiple studies show that funded applicants are at least modestly more productive and more frequently cited after the award as compared to unfunded (Armstrong et al, 1997;Mavis and Katz, 2003;Mahoney et al, 2007;Bornmann et al, 2008bBornmann et al, , 2010Pion and Cordray, 2008;Reinhart, 2009;Campbell et al, 2010; Jacob and Lefgren, 2011a,b;Langfeldt et al, 2012;Robitaille et al, 2015;Van den Besselaar and Sandstrom, 2015;Gush et al, 2017), although some do not (Saygitov, 2014). Interpretation of these results is difficult because it is challenging to dissociate the productivity effect of funding from the validity of the review decision.…”

“…However, while general research funding is related to scientific productivity and knowledge production Rosenbloom et al, 2015) and papers with funding acknowledgments are linked to higher citation counts (Gok et al, 2016), the effect of specific funding on an individual's productivity is not clear; some research looking at ex ante and ex post bibliographic levels for funded applicants show no effect of funding at all (Langfeldt et al, 2012;Robitaille et al, 2015), although it seems the length of time used to capture ex post bibliometric data is an important factor (Van den Besselaar and Sandstrom, 2015). Once again, many of these studies show significant type II errors (where unfunded applicants perform well) and sometimes only limited or no differences are found between funded and unfunded applicants with similar review scores or performance (Bornmann et al, 2008b(Bornmann et al, , 2010Pion and Cordray, 2008;Jacob and Lefgren, 2011a;Van den Besselaar and Sandstrom, 2015;Gush et al, 2017) although some similar comparisons do find differences (Robitaille et al, 2015). These ex post studies are related to the above ex ante results in that some literature has indicated that one of the strongest predictors of future citation performance is prior citation performance (Kaltman et al, 2014;Hutchins et al, 2016).…”

“…Other metrics along the same vein have been used as well, including career satisfaction and faculty positions attained, both of which have been observed to be higher among funded applicants compared to similarly high ex-ante performing unfunded applicants (Hornbostel et al, 2009;Bloch et al, 2014;Van den Besselaar and Sandstrom, 2015). However, while Pion and Ionescu-Pioggia (2003) also found funded applicants of the Burroughs Welcome Career Award were more successful than unfunded in securing faculty positions and in acquiring future NIH funding (Pion and Cordray, 2008), these effects were diminished when adjusted for the ex-ante qualifications of the applicants. Career satisfaction is another variable to be tracked, although only two studies have examined this (Hornbostel et al, 2009;Langfeldt et al, 2012), tracking satisfaction via survey.…”

Evaluating Postgraduate Public Health and Biomedical Training Program Outcomes:

“…One conundrum is individuals are not randomized to training programs but rather select training programs based on their preferences and are, in turn, selected by program directors to join the training program [22]. In the literature review described here, only one study had a comparison group.…”

“…Options for different types of comparison groups are explored in Table 4. If no comparison group is utilized, one needs to be mindful of the effect of time on certain outcomes and how this may bias the results [22]. For example, in research-intensive environments, publication number should increase with duration of career and may not be directly attributable to participation in the training program.…”