Pre-Analysis Plans: An Early Stocktaking

Abstract

Pre-analysis plans (PAPs) have been championed as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze a representative sample of 195 PAPs registered on the Evidence in Governance and Politics (EGAP) and American Economic Association (AEA) registration platforms to assess whether PAPs registered in the early days of pre-registration (2011–2016) were sufficiently clear, precise, and comprehensive to achieve their objective of preventing “fishing” and reducing the scope for post-hoc adjustment of research hypotheses. We also analyze a subset of ninety-three PAPs from projects that resulted in publicly available papers to ascertain how faithfully they adhere to their pre-registered specifications and hypotheses. We find significant variation in the extent to which PAPs registered during this period accomplished the goals they were designed to achieve. We discuss these findings in light of both the costs and benefits of pre-registration, showing how our results speak to the various arguments that have been made in support of and against PAPs. We also highlight the norms and institutions that will need to be strengthened to augment the power of PAPs to improve research credibility and to create incentives for researchers to invest in both producing and policing them.

Reducing bias, increasing transparency, and calibrating confidence with preregistration

“…• Preregistrations often lack detail or are ambiguously specified in medicine 55,[63][64][65] , economics 83,207 , and on the Open Science Framework 109 .…”

“…• Unacknowledged discrepancies between preregistrations and published reports appear to be common in several domains including clinical trials [55][56][57][58][59][60][61] , psychology 208 , and economics 83 .…”

“…Researchers have voiced concern that preregistration devalues or even prevents exploratory analyses 76,77,83,101,224 ; however, this is not intended. Both exploratory (outcome-dependent) analyses and confirmatory (outcome-independent) analyses are important research activities 225,226 with complementary strengths and weaknesses.…”

“…Policy makers (e.g., journal editors) could partner with meta-researchers to formally evaluate different policy options, preferably with randomized trials. Key questions may include (1) does preregistration hamper exploratory research 76,77,83,101,224 , create burdensome bureaucracy 232 , or give research "a superficial veneer of rigor" 70 ? Though preregistration proponents have offered conceptual rebuttals to concerns like these 20,72,127,236 , they will depend on how preregistration is implemented and can be assessed empirically.…”

Tie my hands loosely

“…However, it turns out there is no standard format for PAPs or what ought to be included in them, nor is there much agreement on how stringently to adhere to them. In perhaps the most comprehensive treatment of PAPs in political science, Ofosu and Posner (2021) present evidence from an analysis of PAPs in political science along with a survey of PAP authors. They find there is a large amount of variation in what is included in PAPs.…”