Transparency, Reproducibility, and the Credibility of Economics Research

Abstract

Abstract:There is growing interest in enhancing research transparency and reproducibility in economics and other scientific fields. We survey existing work on these topics within economics, and discuss the evidence suggesting that publication bias, inability to replicate, and specification searching remain widespread in the discipline. We next discuss recent progress in this area, including through improved research design, study registration and pre-analysis plans, disclosure standards, and open sharing of data and materials, drawing on experiences in both economics and other social sciences. We discuss areas where consensus is emerging on new practices, as well as approaches that remain controversial, and speculate about the most effective ways to make economics research more credible in the future. 1 We thank the editor Steven Durlauf and four anonymous referees for useful comments.

The Elasticity of Taxable Income: A Meta-Regression Analysis

“…Finally, since reporting bias influences income elasticities, I propose that empirical research need to agree on reporting standards or even a pre-analysis plan (see Christensen and Miguel (2016) and Slemrod (2016)). Since researchers need to use very sensitive tax return data in order to precisely estimate taxable income elasticities, a replication of their findings is almost impossible.…”

“…My study follows a strand of literature that applies meta-regression analysis (see Christensen and Miguel (2016) for a review). 3 Moreover, this is the first study that relates existing empirical evidence to contextual factors.…”

Improving Data Availability: A New SMJ Initiative

“…A number of journals in economics and political science require that authors of empirical work submit their data and code at the time of article publication or paper submission, with exemptions granted for proprietary and legally restricted data (Christensen & Miguel, ). Compelling the disclosure of data, however, can impede research progress by reducing incentives for scholars to collect and organize unique data in the first place (Christensen & Miguel, ). We believe that this problem is especially relevant to strategic management, where many studies rely on data specific to individual firms, industries, and geographic areas.…”

“…Compelling the disclosure of data would make it more difficult to execute such research programs because other researchers would have ready access to the same data, and authors who collected the data in question would risk being “scooped.” One solution to this problem might be to place an embargo on access to the data by other researchers for a specified period of time. However, the appropriate amount of time during which the original authors should have exclusive access to their data is unclear, and crafting such policies can result in complex and difficult‐to‐implement formulas, such as those that take into account not only the amount of time since publication but also the amount of time involved in collecting the data (Christensen & Miguel, ). These types of policies also place journals in the position of policing disclosure of embargoed data.…”

How Do Rating Agencies’ Decisions Impact Stock Markets? A Meta‐analysis

“…Consequently, the published studies frequently adopt a ‘black box’ approach, limiting the understanding produced and these studies’ reproducibility. This point has been previously underlined (Ioannidis and Doucouliagos, ; Ioannidis et al ., ; Christensen and Miguel, ). Furthermore, Harvey () highlights that, given the competition for publication, there is an incentive for direct or indirect p ‐hacking, which is particularly widespread in economics and financial research.…”