How to Make a Pie: Reproducible Research for Empirical Economics and Econometrics

Abstract

Empirical economics and econometrics (EEE) research now relies primarily on the application of code to datasets. Handling the workflow linking datasets, programs, results and finally manuscript(s) is essential if one wish to reproduce results, which is now increasingly required by journals and institutions. We underline here the importance of "reproducible research" in EEE and suggest three simple principles to follow. We illustrate these principles with good habits and tools, with particular focus on their implementation in most popular software and languages in applied economics.

Toward a FAIR Reproducible Research

“…However, despite an apparent consensus on the general problem, the publication of papers exhorting the scientific community to publish reproducible results, and the dissemination of tools, good practices and courses (Stodden et al, 2013;Gentzkow and Shapiro, 2013;Sandve et al, 2013;Orozco et al, Forthcoming 2021), we still observe considerable weaknesses in both researcher and journal practices, leading to the scarce dissemination of raw scientific materials. 1 Many journals do not have a replication or data and code availability policy.…”

“…Others (Baiocchi, 2007;Orozco et al, Forthcoming 2021) have identified possible organizational improvements at the researcher level. These authors proposed principles to link a paper to its raw components, data and code through a clear workflow, following the original idea of reproducible research documents proposed by Knuth (1984Knuth ( , 1992 and his literate programming approach.…”

“…Researchers should initiate these changes. They now have all the resources needed to improve their individual practices to create more reproducible papers by embedding code and data in a documented and written workflow understandable by others, including researchers' own "future selves" (Gentzkow and Shapiro, 2013;Orozco et al, Forthcoming 2021). They are probably in the process of changing their habits already under the pressure of certain journals and research institutions such as the ERC and NSF.…”

“…We choose to eliminate some specialized journals or journals publishing mainly theoretical work. 19 TA B L E 1 Overview of statistics and economics journal policies in 2020 19 See also Orozco et al (Forthcoming 2021) for the evolution of economic journal replication policies (2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015)(2016)(2017)(2018)(2019).…”

Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge

“…The benefits of computational reproducibility-and increased access to data and code, more generally-have already been articulated many times by researchers in many different fields: archaeology (Marwick 2017), bioinformatics (Mangul et al 2018), cell biology (Grüning et al 2018), computational fluid mechanics (Mesnard and Barba 2017), computer systems research (Collberg and Proebsting 2016), economics (Anderson et al 2008;Koenker and Zeileis 2009;Orozco et al 2018), epidemiology (Coughlin 2017;Peng, Dominici, and Zeger 2006;Shepherd et al 2017), geosciences (Claerbout and Karrenbach 1992;Gil et al 2016;Konkol et al 2019), high-energy physics (Chen et al 2018), hydrology (Hutton et al 2016), mathematical and computational biology (Schnell 2018), machine learning (Hutson 2018;Tatman et al 2018), neuroscience (Crook, Davison, and Figure 1. Fragile Families Challenge data structure.…”