Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies

Abstract

Preregistration is a method to increase research transparency by documenting research decisions on a public, third-party repository prior to any influence by data. It is becoming increasingly popular in all subfields of psychology and beyond. Adherence to the preregistration plan may not always be feasible and even is not necessarily desirable, but without disclosure of deviations, readers who do not carefully consult the preregistration plan might get the incorrect impression that the study was exactly conducted and reported as planned. In this paper, we have investigated adherence and disclosure of deviations for all articles published with the Preregistered badge in Psychological Science between February 2015 and November 2017 and shared our findings with the corresponding authors for feedback. Two out of 27 preregistered studies contained no deviations from the preregistration plan. In one study, all deviations were disclosed. Nine studies disclosed none of the deviations. We mainly observed (un)disclosed deviations from the plan regarding the reported sample size, exclusion criteria and statistical analysis. This closer look at preregistrations of the first generation reveals possible hurdles for reporting preregistered studies and provides input for future reporting guidelines. We discuss the results and possible explanations, and provide recommendations for preregistered research.

Why most psychological research findings are not even wrong

“…For example, Bakker et al (2018) note that even determining the exact number of hypotheses in a preregistration was so difficult that interrater agreement on this variable was as low as 14%. Similarly, Claesen et al (2021) state that '[a]ssessing the adherence of the published studies to the preregistration plans proved to be a far from trivial task' because 'neither the preregistration plans nor the published studies were written in sufficient detail for a fair comparison' (p. 4), and van den Akker (2021) remarks that '(r)esearchers are very bad at clearly laying out hypotheses in preregistrations (and in papers)' (p. 31).…”

The Effect of Chronic Exercise on Energy and Fatigue States: A Systematic Review and Meta-Analysis of Randomized Trials

“…All items in this protocol correspond with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols Statement (PRISMA-P; Moher et al, 2016 ). Corresponding to the PRISMA guidelines ( Shamseer et al, 2015 ; Moher et al, 2016 ), our review protocol was registered with the Open Science Framework on 9 March 2021 ( Claesen et al, 2021 ). Registration number and link: 10.17605/OSF.IO/RQ82B .…”

Why most psychological research findings are not even wrong

“…For example, Bakker et al (2018) noted that even determining the exact number of hypotheses in a given preregistration had been so difficult that inter-rater agreement on this variable was as low as 14%. Similarly, Claesen, Gomes, Tuerlinckx, and Vanpaemel (2021) stated that '[a]ssessing the adherence of the published studies to the preregistration plans proved to be a far from trivial task' because 'neither the preregistration plans nor the published studies were written in sufficient detail for a fair comparison' (p. 4), and van den Akker (2021) remarked that '(r)esearchers are very bad at clearly laying out hypotheses in preregistrations (and in papers)' (p. 31).…”