Stagnation and Scientific Incentives

Abstract

We are grateful for support from the National Institute on Aging through program project grant P01 AG039347. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications.

What Can We Do to Improve Peer Review in NLP?

“…Novel approaches. This sounds almost absurd, but scientific peer review is systematically biased towards unobjectionable (rather than novel) work (Church, 2005(Church, , 2020Smith, 2010;Bhattacharya and Packalen, 2020). A reviewer faced with evaluating a completely new idea without prior art has to make a more difficult call than one for a paper with clear predecessors and a leaderboard table, and is more likely to fall back to one of the heuristics.…”

“…More recently, ELMo (Peters et al, 2018) received low scores from ICLR reviewers and was resubmitted to NAACL to win the award there. Realistic paper merit distribution, adapted from Anderson (2009) Paper merit distribution, with which peer review could be reliable As for paper impact, it is distinct from its scientific merit (Bhattacharya and Packalen, 2020), and strongly depends on completely orthogonal factors: how niche is the topic, how much promotion was done, whether the paper offers room to innovate with a low entry barrier 2 (Anderson, 2009). What we could realistically expect from peer review is to reject the papers with obvious methodology flaws, and turn the spotlight on the ideas which would be beneficial for the field to discuss.…”

Political Failures in Innovation Policy: A Cautionary Note

“…In the future, private firms are expected to spend less and less on basic research, which could decrease future growth rates, even though the effects might currently not be acute (Arora et al, 2015;Färnstrand Damsgaard et al, 2017). The growth in productivity and innovation has recently been decelerating, sparking an intense debate about the future of innovation and growth (Cowen, 2011;Gordon, 2016;Bloom et al, 2017;Bhattacharya and Packalen, 2020). Indeed, some scholars describe the current lack of growth as a form of secular stagnation, predicting low levels of long-run growth (Summers, 2015).…”

Evaluation and innovation: An antagonistic pair?

“…Huebner, 2005). Some have argued that the increasing reliance on quantitative evaluation is one of the factors that has contributed to that decline in recent decades (see Bhattacharya and Packalen, 2020). Yet regardless of whether evaluation practices are directly implicated as a factor, the evidence of innovation decline ought to make us wary of structures and practices that reinforce the inevitable tendencies towards normalization.…”