Science Deserves Better: The Imperative to Share Complete Replication Files

Abstract

In April 2013, a controversy arose when a working paper (Herndon, Ash, and Pollin 2013) claimed to show serious errors in a highly cited and influential economics paper by Carmen Reinhart and Kenneth Rogoff (2010). The Reinhart and Rogoff paper had come to serve as authoritative evidence in elite conversations (Krugman 2013) that high levels of debt, especially above the “90 percent [debt/GDP] threshold” (Reinhart and Rogoff 2010, 577), posed a risk to economic growth. Much of the coverage of this controversy focused on an error that was a “perfect made-for-TV mistake” (Stevenson and Wolfers 2013) involving a simple error in the formula used in their Excel calculations. The real story here, however, is that it took three years for this error and other issues to be discovered because replication files were not publicly available, nor were they provided to scholars when asked. If professional norms or theAmerican Economic Reviewhad required that authors publish replication files, this debate would be advanced by three years and discussions about austerity policies would have been based on a more clear-sighted appraisal of the evidence.

Laying the Foundations for Scientometric Research

“…Finally, the field of social sciences has recently come under serious scrutiny, given fraudulent and questionable research practices (see Bhattacharjee, 2013;McNutt, 2015). We use common tools of data science to make our work fully open and reproducible, which is regarded as an essential step in addressing quality concerns in the social sciences (see Dafoe, 2014;King, 1995;Yong, 2012). To our knowledge, this study is the first fully reproducible publication in a social work journal.…”

Tool-driven Revolutions in Archaeological Science

“…For example, the journal Nature Neuroscience requires authors 'to make the code that supports the generation of key figures in their manuscript available for review' (Editors 2017). Although we are not aware of attempts to quantify this change in the same way we have for archaeology, we have observed a number of recent publications that describe and recommend code sharing in statistics (Baumer et al 2014), genome biology (Markowetz 2015), computational biology (Sandve et al 2013), hydrology (Slater et al 2019), biostatistics (Peng 2009), computer science (Peng 2011;Mitchell et al 2012), applied mathematics (LeVeque et al 2012), speech science (Abari 2012), political science (King 1995;Dafoe 2014), and the social sciences generally (Miguel et al 2014). As part of this growing interest in using code for research we also see manifestos aimed at researchers doing any kind of quantitative work (B.…”

Reproducibility: A Primer on Semantics and Implications for Research

“…3 Scientists across numerous disciplines have expressed serious concerns and engaged in both philosophical and practical discussions about reproducible research. Disciplines represented in these discussions include biology, 4 biomedical 5 and preclinical research, 6,7 business and organizational studies, [8][9][10][11][12] computational sciences, 13,14 drug discovery, 15 economics, 16,17 education, [18][19][20][21][22] epidemiology and statistics, [23][24][25] genetics, 26 immunology, 27 policy research, 28 political science, [29][30][31][32][33][34][35][36] psychology, 29,[37][38][39][40][41][42][43] and sociology. 44 As a case in point: research indicates that more than half of psychology studies fail reproducibility tests.…”