Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰

Abstract

The scientific community is engaged in an active debate on the value of its peer-review system. Does peer review actually serve the role we envision for it-that of helping government agencies predict what ideas have the best chance of contributing to scientific advancement? Many federal agencies use a two-step review process that includes programmatic discretion in selecting awards. This process allows us to determine whether success in a future independent scientific-research career is more accurately predicted by peer-review recommendations or discretion by program staff and institute leaders. Using data from a prestigious training program at the National Institute of Health (NIH), the Ruth L. Kirschstein National Research Service Award (NRSA), we provide evidence on the efficacy of peer review. We find that, despite all current claims to the contrary, the existing peer-review system works as intended. It more closely predicts high-quality science and future research independence than discretion. We discover also that regression discontinuity, the econometric method typically used to examine the effect of scientific funding, does not fit many scientific-funding models and should only be used with caution when studying federal awards for science.

Applied usage and performance of statistical matching in bibliometrics: The comparison of milestone and regular papers with multiple measurements of disruptiveness as an empirical example

“…Previous studies compare the results of at least two techniques (Ginther & Heggeness, 2020) or combine different matching techniques (Farys & Wolbring, 2017). We would like to encourage researchers to follow these examples and we suggest to apply more than one algorithm and compare results.…”

“…A few earlier studies by Mutz and Daniel (2012), Mutz, Wolbring, and Daniel (2017), Farys and Wolbring (2017), and Ginther and Heggeness (2020) have demonstrated how useful matching techniques are for scientometric studies. For example, Mutz, Wolbring, and Daniel (2017) have used the technique to investigate the effect of assigning the label "very important paper" to papers published in the journal Angewandte Chemie -International Edition.…”

Self-Citation, Cumulative Advantage, and Gender Inequality in Science

“…By sampling on winners of just these four prestigious, merit-based awards, we are confident that our sample is relatively homogenous in terms of scholarly potential, a difficult to observe characteristic. That said, we are aware that a recent study (Ginther and Heggeness 2020) raises the issue of administrative discretion in allocating supposedly merit-based awards in science. They find that discretion is widely used by program officers when allocating the National Institutes of Health (NIH) F32 National Research Service Award, a grant given to "high-potential, earlycareer scientists" in the biomedical sciences.…”

“…They find that discretion is widely used by program officers when allocating the National Institutes of Health (NIH) F32 National Research Service Award, a grant given to "high-potential, earlycareer scientists" in the biomedical sciences. Although the award is undoubtedly still a selective prize, Ginther and Heggeness (2020) show that award winners include more applicants "reached for" through discretionary decisions compared with applicants funded based on high peer review scores alone. Unfortunately, we do not have access to reviewer scores from the private fellowships on which our sample is based and thus cannot directly examine the reward allocation process of these private fellowships à la Ginther and Heggeness (2020).…”

“…Although the award is undoubtedly still a selective prize, Ginther and Heggeness (2020) show that award winners include more applicants "reached for" through discretionary decisions compared with applicants funded based on high peer review scores alone. Unfortunately, we do not have access to reviewer scores from the private fellowships on which our sample is based and thus cannot directly examine the reward allocation process of these private fellowships à la Ginther and Heggeness (2020). We strongly suspect, however, that heterogeneity in scholarly potential is limited in our sample compared with the F32 award given that our fellowships are controlled by private foundations and the total number of awards per year is far lower compared with the F32.…”