Flexible Yet Fair: Blinding Analyses in Experimental Psychology

Dutilh, Gilles; Sarafoglou, Alexandra; Wagenmakers, Eric‐Jan

doi:10.31234/osf.io/d79r8

Cited by 11 publications

(14 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 1. This method of blinding the data involves the data handler switching the labels of key hypothesis-testing variables with other variables so as to reduce the potential for bias in the analysis process (Dutilh et al, 2019). …”

mentioning

confidence: 99%

A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect

et al. 2021

Self Cite

View full text Add to dashboard Cite

We conducted a preregistered multilaboratory project ( k = 36; N = 3,531) to assess the size and robustness of ego-depletion effects using a novel replication method, termed the paradigmatic replication approach. Each laboratory implemented one of two procedures that was intended to manipulate self-control and tested performance on a subsequent measure of self-control. Confirmatory tests found a nonsignificant result ( d = 0.06). Confirmatory Bayesian meta-analyses using an informed-prior hypothesis (δ = 0.30, SD = 0.15) found that the data were 4 times more likely under the null than the alternative hypothesis. Hence, preregistered analyses did not find evidence for a depletion effect. Exploratory analyses on the full sample (i.e., ignoring exclusion criteria) found a statistically significant effect ( d = 0.08); Bayesian analyses showed that the data were about equally likely under the null and informed-prior hypotheses. Exploratory moderator tests suggested that the depletion effect was larger for participants who reported more fatigue but was not moderated by trait self-control, willpower beliefs, or action orientation.

show abstract

mentioning

confidence: 99%

A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some researchers might also prefer alternative methods to preregistration. One of these alternatives that allows for more flexibility while still safeguarding the confirmatory status of the research is analysis blinding (MacCoun, 2020;MacCoun & Perlmutter, 2015, 2018Dutilh, Sarafoglou, & Wagenmakers, 2019). With analysis blinding, researchers are in principle not required to write a preregistration document.…”

Section: Discussionmentioning

confidence: 99%

A survey on how preregistration affects the research workflow: Better science but more work

Sarafoglou¹,

Kovács²,

Bakos³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The preregistration of research protocols and analysis plans is a main reform innovation to counteract confirmation bias in the social and behavioral sciences. While theoretical reasons to preregister are frequently discussed in the literature, the individually experienced advantages and disadvantages of this method remain largely unexplored. The goal of this exploratory study was to identify the benefits and challenges of preregistration from the researcher's perspective. To this aim, we surveyed 355 researchers, 299 of whom had used preregistration in their own work. The researchers indicated the experienced or expected effects of preregistration on their workflow. The results show that experiences and expectations are mostly positive. Researchers in our sample believe that implementing preregistration improves or is likely to improve the quality of their projects, and that preregistration makes it easier to avoid questionable research practices. Criticism of preregistration is primarily related to the increase in work-related stress and the overall duration of the project. The majority of researchers with experience in preregistration reported that the benefits outweigh the challenges. However, the majority of researchers without preregistration would not consider preregistration for future projects or recommend the practice to colleagues. Our interpretation of the results is that preregistration can have positive side-effects as it adds an extra preparatory step in researchers' workflow, thus requiring researchers to think through the theoretical and practical aspects of their project.

show abstract

“…This blinding reduces the risk of inadvertent bias, by which investigators find effects where none actually exist. 19 When a control condition is used, it should be an active control wherein the control group participants receive an intervention similar in size, scope, and resources to the main intervention being tested. [20][21][22] Active controls reduce the risk that an apparent improvement following an intervention is simply due to something happening to participants.…”

Section: Design Choicesmentioning

confidence: 99%

Is There a Replication Crisis in Medical Education Research?

2021

View full text Add to dashboard Cite

Scholars are increasingly aware that studies-across many disciplinescannot be replicated by independent researchers. Here, the authors describe how medical education research may be vulnerable to this "replication crisis," explain how researchers can act together to reduce risks, and discuss the positive steps that can increase confidence in research findings. Medical education research contributes to policy and influences practitioner behavior. Findings that cannot be replicated suggest that the original research was not credible. This risk raises the possibility that unhelpful or even harmful changes to medical education have been implemented as a result of research that appeared defensible but was not. By considering these risk factors, researchers can increase the likelihood that studies are generating credible results. The authors discuss and provide examples of 6 factors that may endanger the replicability of medical education research: (1) small sample sizes, (2) small effect sizes, (3) exploratory designs, (4) flexibility in design choices, analysis strategy, and outcome measures, (5) conflicts of interest, and ( 6) very active fields with many competing research teams. Importantly, medical education researchers can adopt techniques used successfully elsewhere to improve the rigor of their investigations. Researchers can improve their work through better planning in the development stage, carefully considering design choices, and using sensible data analysis. The wider medical education community can help by encouraging higher levels of collaboration among medical educators, by routinely evaluating existing educational innovations, and by raising the prestige of replication and collaborative medical education research. Medical education journals should adopt new approaches to publishing. As medical education research improves, so too will the quality of medical education and patient care.

show abstract

Flexible Yet Fair: Blinding Analyses in Experimental Psychology

Cited by 11 publications

References 47 publications

A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect

A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect

A survey on how preregistration affects the research workflow: Better science but more work

Is There a Replication Crisis in Medical Education Research?

Contact Info

Product

Resources

About