Replication studies in economics—How many and which papers are chosen for replication, and why?

Mueller‐Langer, Frank; Fecher, Benedikt; Harhoff, Dietmar; Wagner, Gert G.

doi:10.1016/j.respol.2018.07.019

Cited by 98 publications

(75 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In spite of these caveats, we conclude that Registered Reports are promoting reproducibility, transparency and self-correction across a wide range of disciplines, and may help reshape how society evaluates research and researchers.1 After more than a decade of meta-research and debate, the life and social sciences are well in the midst of a credibility revolution 1-3 . Faced with evidence of publication bias 4-7 , hindsight bias and selective reporting [8][9][10][11][12][13][14][15] , insufficient sample sizes 16 , inadequate data sharing 17 , and suboptimal rates of both attempted 18,19 and successful replication 20-22 , researchers from across a broad range of fields are unifying around a core mission to improve reproducibility and transparency. In doing so, the deeper aim of the openness agenda is to stimulate cultural reform, aligning the needs of individual scientists with the needs of the scientific community 23 .As scientists and policymakers grapple with the causes of irreproducibility, it has become clear that one of its main drivers is the so-called results paradox .…”

mentioning

confidence: 99%

The past, present, and future of Registered Reports

Chambers¹,

Tzavella²

2020

Preprint

103

View full text Add to dashboard Cite

Registered Reports are a form of empirical journal article in which study proposals are peer reviewed and pre-accepted before research is undertaken. By deciding which articles are published based on the question, theory, and proposed methods, Registered Reports offer a powerful remedy for a range of reporting and publication biases. Here we reflect on the history, progress and future prospects of the Registered Reports initiative, and also offer practical guidance for authors, reviewers, and editors encountering the format for the first time. While the key ingredients of pre-study review and results-blind acceptance are far from novel – and are already adopted independently in a variety of contexts – Registered Reports are the first mechanism to combine them into a mainstream policy that has won appeal with multiple stakeholders in the research process. We review early evidence that Registered Reports are working as intended, while at the same acknowledging that they are not a universal solution for irreproducibility. We also consider how the policies and practices surrounding Registered Reports are changing, or must change in future, to address limitations and adapt to new challenges. In spite of these caveats, we conclude that Registered Reports are promoting reproducibility, transparency and self-correction across a wide range of disciplines, and may help reshape how society evaluates research and researchers.

show abstract

mentioning

confidence: 99%

The past, present, and future of Registered Reports

Chambers¹,

Tzavella²

2020

Preprint

103

View full text Add to dashboard Cite

show abstract

“…Despite a general consensus among economists regarding the importance of replication, it remains largely “an ideal to be professed but not practiced” (Mueller‐Langer et al ). Incentives for individual researchers to replicate a project or to have their own work replicated are low or missing entirely.…”

Section: Dozen Thingsmentioning

confidence: 99%

“…46 Within the same framework, Coffman and Niederle (2015) argue that even the most inaccurate beliefs can be corrected within three to five replications. 47 Despite a general consensus among economists regarding the importance of replication, it remains largely "an ideal to be professed but not practiced" (Mueller-Langer et al 2019). Incentives for individual researchers to replicate a project or to have their own work replicated are low or missing entirely.…”

Section: Replicate Early and Oftenmentioning

confidence: 99%

“…Moreover, journals may be reluctant to publish replication studies for fear of not receiving enough citations (Duvendack, Palmer-Jones, and Reed 2017). Indeed, according to a survey by Mueller-Langer et al (2019), from 1974 to 2014 less than 0.1% of publications in the top-50 economics journals were replications. Given the difficulties of publishing a "mere replication," conducting an extension study where the control treatment replicates a previous finding is often a more attractive alternative.…”

Section: Replicate Early and Oftenmentioning

confidence: 99%

See 1 more Smart Citation

The Dozen Things Experimental Economists Should Do (More of)

Czibor¹,

Jimenez-Gomez²,

List

2019

Southern Economic Journal

View full text Add to dashboard Cite

What was once broadly viewed as an impossibility—learning from experimental data in economics—has now become commonplace. Governmental bodies, think tanks, and corporations around the world employ teams of experimental researchers to answer their most pressing questions. For their part, in the past two decades academics have begun to more actively partner with organizations to generate data via field experimentation. Although this revolution in evidence‐based approaches has served to deepen the economic science, recently a credibility crisis has caused even the most ardent experimental proponents to pause. This study takes a step back from the burgeoning experimental literature and introduces 12 actions that might help to alleviate this credibility crisis and raise experimental economics to an even higher level. In this way, we view our “12 action wish list” as discussion points to enrich the field.

show abstract

“…Moreover, very few replication studies are published in top economics journals. A recent study finds that from 1974 to 2014 only 130 replication studies were published in the top 50 economics journals [8]. This translates to a share of replications over the total number of published studies of about 0.1%.…”

Section: Replications and Data Availability Policiesmentioning

confidence: 99%