The effect of public funding on research output: the New Zealand Marsden Fund

Abstract

We estimate the impact of participating in the NZ Marsden Fund on research output trajectories, by comparing the subsequent performance of funded researchers to those who submitted proposals but were not funded. We control for selection bias using the evaluations of the proposals generated by the grant selection process. We carry out the analysis in two data frames. First we consider the researcher teams behind 1263 second-round proposals submitted 2003-2008, and look at the post-proposal publication and citation performance of the team as a whole, as a function of pre-proposal performance, the ranking of the proposal by the panel, and the funding. This estimation does not deal with individual researchers' multiple proposals and funding over time. To disentangle these effects, we consider the 1500 New Zealand researchers who appeared on any of these proposals, and estimate a model predicting annual individual performance as a function of previous performance, recent proposal activity, ranking of any recent proposals, and funding received through recent proposals. Overall, we find that funding is associated with a 6-15% increase in publications and a 22-26% increase in citation-weighted papers for research teams. For individuals, funding is associated with a 3-5% increase in annual publications, and a 5-8% increase in citation-weighted papers for 5 years after grant; however, the lag structure and persistence of this effect post-grant is difficult to pin down. Surprisingly, we find no systematic evidence that the evaluation of proposals by the Marsden system is predictive of subsequent success. We conclude that the Marsden Fund is modestly successful in increasing scientific performance, but that the selection process does not appear to be effective in discriminating among second-round proposals in terms of their likely success.

External Tests of Peer Review Validity Via Impact Measures

“…Multiple studies show that funded applicants are at least modestly more productive and more frequently cited after the award as compared to unfunded (Armstrong et al, 1997;Mavis and Katz, 2003;Mahoney et al, 2007;Bornmann et al, 2008bBornmann et al, , 2010Pion and Cordray, 2008;Reinhart, 2009;Campbell et al, 2010; Jacob and Lefgren, 2011a,b;Langfeldt et al, 2012;Robitaille et al, 2015;Van den Besselaar and Sandstrom, 2015;Gush et al, 2017), although some do not (Saygitov, 2014). Interpretation of these results is difficult because it is challenging to dissociate the productivity effect of funding from the validity of the review decision.…”

“…In fact, several other studies using normalized and unnormalized citation impact measures also suggested a moderate correlation (Berg, 2011;Gallo et al, 2014). When NIH data were reanalyzed without using budget normalized citation impact, a moderate correlation was observed .A few other studies have found no correlation between scores and citation impact, although one was a very small sample (Scheiner and Bouchie, 2013) and the other was from the second round of review, so the level of quality across these projects was already very high (Gush et al, 2017). In fact, similar results were found with NIH data (same data set as used by Li and Agha, 2015); if the poorer scoring applications were removed from the analysis to reflect current funding rates, correlations between output and review scores disappeared (Fang et al, 2016).…”

“…However, while general research funding is related to scientific productivity and knowledge production Rosenbloom et al, 2015) and papers with funding acknowledgments are linked to higher citation counts (Gok et al, 2016), the effect of specific funding on an individual's productivity is not clear; some research looking at ex ante and ex post bibliographic levels for funded applicants show no effect of funding at all (Langfeldt et al, 2012;Robitaille et al, 2015), although it seems the length of time used to capture ex post bibliometric data is an important factor (Van den Besselaar and Sandstrom, 2015). Once again, many of these studies show significant type II errors (where unfunded applicants perform well) and sometimes only limited or no differences are found between funded and unfunded applicants with similar review scores or performance (Bornmann et al, 2008b(Bornmann et al, , 2010Pion and Cordray, 2008;Jacob and Lefgren, 2011a;Van den Besselaar and Sandstrom, 2015;Gush et al, 2017) although some similar comparisons do find differences (Robitaille et al, 2015). These ex post studies are related to the above ex ante results in that some literature has indicated that one of the strongest predictors of future citation performance is prior citation performance (Kaltman et al, 2014;Hutchins et al, 2016).…”

How gender determines the way we speak about professionals

“…The Matthew Effect (17) refers to a rich-get-richer phenomenon in science and academia, such that "recognition is awarded partly on the basis of past recognition" (18). This notion was first suggested in 1968 and has since been shown repeatedly in various disciplines (19)(20)(21)(22)(23)(24). For instance, reviewers are more likely to accept papers by famous authors when they know the authors' identity than when they are blind to it (22).…”

Pūtahitanga: the intersection of western science and mātauranga Māori in the context of Aotearoa New Zealand’s physical oceanography

“…Few data are available on Māori participation in mathematics or the physical sciences. The Marsden Fund run by the Royal Society Te Apārangi (Gush et al 2018) makes available numbers on proposal success rates indicating Māori success rates are of the order of 10% ( Figure 2) although oceanography projects are rare. For school ages, international surveys of competency point to a correlation with socio-economic well-being when quantifying low equity outcomes (Caygill et al 2016;May et al 2016).…”