Is Preregistration Worthwhile?

Szöllősi, Ágnes; Kellen, David; Navarro, Daniel J.; Shiffrin, Richard M.; Rooij, Iris van; Zandt, Trisha Van; Donkin, Chris

doi:10.1016/j.tics.2019.11.009

Cited by 117 publications

(103 citation statements)

References 12 publications

Supporting

Mentioning

101

Contrasting

Unclassified

Order By: Relevance

“…Indeed, given the almost dizzying array of choices necessary for ESM research, being able to refer back to a locked, timestamped record of these choices is advantageous. There are also some threats to reproducibility that (pre-)registration does not solve, for example, issues of weak theorizing and poor correspondence between theories and the statistical models that are supposed to map onto them (Szollosi et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)

Lafit

Achterhof

Hiekkaranta

et al. 2019

Preprint

View full text Add to dashboard Cite

A growing interest in understanding complex and dynamic psychological processes as they occur in everyday life has led to an increase in studies using Ambulatory Assessment techniques, including the Experience Sampling Method (ESM) and Ecological Momentary Assessment (EMA). There are, however, numerous “forking paths” and researcher degrees of freedom, even beyond those typically encountered with other research methodologies. Whilst a number of researchers working with ESM techniques are actively engaged in efforts to increase the methodological rigor and transparency of such research, currently, there is little routine implementation of open science practices in ESM research. In the current paper, we discuss the ways in which ESM research is especially vulnerable to threats to transparency, reproducibility and replicability. We propose that greater use of (pre-)registration, a cornerstone of open science, may address some of these threats to the transparency of ESM research. (Pre-)registration of ESM research is not without challenges, including model selection, accounting for potential model convergence issues and the use of pre-existing datasets. As these may prove to be significant barriers to (pre-)registration for ESM researchers, we also discuss ways of overcoming these challenges and of documenting them in a (pre-)registration. A further challenge is that current general templates do not adequately capture the unique features of ESM. Here we present a (pre-)registration template for ESM research, adapted from the original Pre-Registration Challenge (Mellor et al., 2019) and pre-registration of pre-existing data (van den Akker et al., 2020) templates, and provide examples of how to complete this.

show abstract

Section: Discussionmentioning

confidence: 99%

Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)

Lafit

Achterhof

Hiekkaranta

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…These negative theoretical qualities will detract from the plausibility of the hypotheses and the credibility of the research conclusions. Importantly, poor quality theorizing can be identified and taken into account by readers in the absence of preregistration (see also Szollosi & Donkin, 2019). Hence, preregistration does not provide a credibility advantage in this respect.…”

Section: Identifying Harkingmentioning

confidence: 99%

“…It is also important to recognize that low replication rates have been attributed to more than just questionable research practices. For example, low replication rates have been attributed to (a) insufficiently stringent evidence thresholds (Benjamin et al, 2018), (b) insufficiently lenient evidence thresholds (Devezer et al, 2020), (c) poor measurement (e.g., Loken & Gelman, 2017), (d) model misspecification (Devezer et al, 2020), (e) low power (e.g., Rossi, 1990), (f) poor theory (e.g., Oberauer & Lewandowsky, 2019;Szollosi & Donkin, 2019), (g) an underappreciation of the influence of hidden moderators (Rubin, 2019b), and (h) errors in substantive inference (Jussim et al, 2016;Rubin, 2017b, p. 274). Hence, it is unclear whether preregistration is targeting the right set of issues to increase replication rates.…”

Section: Improving Replication Ratesmentioning

confidence: 99%

“…Several previous articles have questioned the added value of preregistration (e.g., Baron, 2018;Coffman & Niederle, 2015;Devezer et al, 2020;Ledgerwood, 2018;Lewandowsky, 2019;Navarro, 2019;Oberauer & Lewandowsky, 2019;Szollosi & Donkin, 2019;Szollosi et al, 2020). However, none of these articles have distinguished between historical and contemporary transparency, and none have considered the full range of advantages that are supposed to be associated with preregistration.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Does preregistration improve the credibility of research findings?

Rubin¹

2020

Preprint

View full text Add to dashboard Cite

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers’ biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration’s historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.

show abstract

“…Finally, keep in mind that a preregistered study is not necessarily a high‐quality study: Preregistration does not automatically improve a flawed research question or proposed methodology (e.g., Szollosi et al., 2019).…”

Section: Introductionmentioning

confidence: 99%

Preregistration in infant research—A primer

Havron¹,

Bergmann

Tsuji

2020

Infancy

View full text Add to dashboard Cite

Preregistration, the act of specifying a research plan in advance, is becoming more common in scientific research. Infant researchers contend with unique problems that might make preregistration particularly challenging. Infants are a hard‐to‐reach population, usually yielding small sample sizes, they can only complete a limited number of trials, and they can be excluded based on hard‐to‐predict complications (e.g., parental interference, fussiness). In addition, as effects themselves potentially change with age and population, it is hard to calculate an a priori effect size. At the same time, these very factors make preregistration in infant studies a valuable tool. A priori examination of the planned study, including the hypotheses, sample size, and resulting statistical power, increases the credibility of single studies and adds value to the field. Preregistration might also improve explicit decision making to create better studies. We present an in‐depth discussion of the issues uniquely relevant to infant researchers, and ways to contend with them in preregistration and study planning. We provide recommendations to researchers interested in following current best practices.

show abstract

Is Preregistration Worthwhile?

Cited by 117 publications

References 12 publications

Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)

Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)

Does preregistration improve the credibility of research findings?

Preregistration in infant research—A primer

Contact Info

Product

Resources

About