Research Registries: Facts, Myths, and Possible Improvements

Abstract

The past few decades have ushered in an experimental revolution in economics whereby scholars are now much more likely to generate their own data. While there are virtues associated with this movement, there are concomitant difficulties. Several scientific disciplines, including economics, have launched research registries in an effort to attenuate key inferential issues. This study assesses registries both empirically and theoretically, with a special focus on the AEA registry. We find that over 90% of randomized control trials (RCTs) in economics do not register, only 50% of the RCTs that register do so before the intervention begins, and the majority of these preregistrations are not detailed enough to significantly aid inference. Our empirical analysis further shows that using other scientific registries as aspirational examples is misguided, as their perceived success in tackling the main issues is largely a myth. In light of these facts, we advance a simple economic model to explore potential improvements. A key insight from the model is that removal of the (current) option to register completed RCTs could increase the fraction of trials that register. We also argue that linking IRB applications to registrations could further increase registry effectiveness.

Preregistration and Credibility of Clinical Trials*

“…However, studies with a complete pre-analysis plan appear significantly less p-hacked. Abrams, Libgober, and List (2023) present evidence that the uptake and quality of registrations in the AEA Registry and ClinicalTrials.gov are relatively poor and find that preregistration at the AEA registry does not impact indications of p-hacking in a randomly selected sample of economics RCTs. Moreover, they explore policies that could increase registry effectiveness in a theoretical model.…”

Evidence on Research Transparency in Economics

“…My goal in this piece is not to rehash the ongoing debates about potential benefits of adopting preregistration, and whether they justify the up-front costs. For an overview of these debates, the reader can turn to Olken (2015) and Coffman and Niederle (2015): for additional views, recent starting points are Christensen, Freese, and Miguel (2019); Duflo et al (2020); and Abrams, Libgober, and List (2020). Rather I will briefly sketch the parameters of the existing debate and then devote my attention to newly emerging evidence on the real-world practice and implementation.…”

“…It remains possible that this situation has improved since their data from 2016, but updated research could document how the use of pre-analysis plans has evolved over time. Abrams, Libgober, and List (2020) carry out a related audit of the pre-analysis plans listed on the AEA registry. They point out that norms regarding registration vary considerably even across experimental fields, with high rates among economists conducting field experiments but far lower levels among those carrying out lab experiments.…”

“…In the area of pre-analysis plans, Laitin et al's (2020) plan to Report All Results Efficiently (RARE) proposes to make it standard practice for authors to post all results related to their pre-analysis plans on public study registries, in a so-called "pre-analysis plans report," even if those finding never make their way into a published paper (a proposal related to some ideas developed in parallel in Duflo et al 2020 andAbrams, Libgober, andList 2020). This step would allow searches of study registries to yield far more complete evidence on work that has been carried out on a certain topic to date, leading to improved meta-analysis as well as more informed choices for scholars launching new projects.…”

A Practical Guide to Registered Reports for Economists

“…Ofosu and Posner (2021) show that pre-analysis plans are often not written or used in a way that allow them to solve the issues they are aimed to address. Similarly, Abrams et al (2020) analyze pre-registrations in experimental economics and conclude that the majority of these pre-registrations are not detailed enough to address the concerns about inference. Fourth, RRs create better incentives for researchers as compared to pre-registration: in-principle acceptance (IPA) increases the likelihood of innovative approaches, as researchers know that high-risk, high-reward protocols will be published when they receive an IPA, regardless of the outcome.…”