Reducing bias, increasing transparency, and calibrating confidence with preregistration

Hardwicke, Tom E; Wagenmakers, Eric‐Jan

doi:10.31222/osf.io/d7bcu

Cited by 9 publications

(9 citation statements)

References 150 publications

(201 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On a more general note, it should be borne in mind that that preregistration has a variety of benefits that help increase scientific rigor, but it is not a panacea (as argued by [87]). Therefore, preregistration should ideally be used together with complementary open science practices increasing transparency and confidence (see [87] for an overview).…”

Section: Limitationsmentioning

confidence: 99%

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Spitzer

Mueller²

2023

PLoS ONE

View full text Add to dashboard Cite

Background Preregistration, the open science practice of specifying and registering details of a planned study prior to knowing the data, increases the transparency and reproducibility of research. Large-scale replication attempts for psychological results yielded shockingly low success rates and contributed to an increasing demand for open science practices among psychologists. However, preregistering one’s studies is still not the norm in the field. Here, we conducted a study to explore possible reasons for this discrepancy. Methods In a mixed-methods approach, we conducted an online survey assessing attitudes, motivations, and perceived obstacles with respect to preregistration. Respondents (N = 289) were psychological researchers that were recruited through their publications on Web of Science, PubMed, PSYNDEX, and PsycInfo, and preregistrations on OSF Registries. Based on the theory of planned behavior, we predicted that positive attitudes (moderated by the perceived importance of preregistration) as well as a favorable subjective norm and higher perceived behavioral control positively influence researchers’ intention to preregister (directional hypothesis 1). Furthermore, we expected an influence of research experience on attitudes and perceived motivations and obstacles regarding preregistration (non-directional hypothesis 2). We analyzed these hypotheses with multiple regression models and included preregistration experience as a control variable. Results Researchers’ attitudes, subjective norms, perceived behavioral control, and the perceived importance of preregistration significantly predicted researchers’ intention to use preregistration in the future (see hypothesis 1). Research experience influenced both researchers’ attitudes and their perception of motivations to preregister, but not the perception of obstacles (see hypothesis 2). Descriptive reports on researchers’ attitudes, motivations and obstacles regarding preregistration are provided. Discussion Many researchers had already preregistered and had a rather positive attitude toward preregistration. Nevertheless, several obstacles were identified that may be addressed to improve and foster preregistration.

show abstract

Section: Limitationsmentioning

confidence: 99%

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Spitzer

Mueller²

2023

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…If space in the paper is lim-ited, these details should be given in a footnote or endnote, or in a supplementary technical report. Ideally, strategies for identifying careless respondents are already detailed during preregistration of the study (see Hardwicke & Wagenmakers, 2022, for an introduction to preregistration).…”

Section: Guidelines For Researchersmentioning

confidence: 99%

Open Science Perspectives on Machine Learning for the Identification of Careless Responding: A New Hope or Phantom Menace?

Alfons¹,

Welz²

2022

Preprint

View full text Add to dashboard Cite

While machine learning has gained substantial attention and popularity in many scientific fields, it is largely unexplored for the detection of careless responding in survey data. On the one hand, machine learning algorithms can be highly powerful tools due to their flexibility. On the other hand, science based on machine learning has been criticized in the literature for a lack of reproducibility. We assess the potential and the pitfalls of machine learning approaches for identifying careless respondents from an open science perspective. In particular, we discuss possible sources of reproducibility issues when applying machine learning in the context of careless responding, and we give practical guidelines on how to avoid them. Furthermore, we illustrate the high potential of machine learning for the identification of careless respondents in a proof-of-concept simulation experiment. In a longer-term perspective, we elaborate on how open science practices such as sharing code and data may elevate this potential to an even higher level in the future by establishing benchmark data sets in which respondents are labeled with respect to carelessness.

show abstract

“…Preregistration is expected to effectively prevent undisclosed analytical flexibility, including (unintentional) p-hacking, "cherry-picking" of results, and certain forms of HARKing (Hypothesizing After The Results Are Known) 10,[22][23][24][25][26] . Preregistrations are made publicly available before data collection or can be embargoed for a specific period of time on websites like the Open Science Framework (OSF; osf.io).…”

Section: Introductionmentioning

confidence: 99%

Using preregistration as a tool for transparent fNIRS study design

Schroeder¹,

Artemenko²,

Kosie

et al. 2022

Preprint

View full text Add to dashboard Cite

Significance: The expansion of fNIRS methodology and analysis tools give rise to various design and analytical decisions researchers have to make. Several recent efforts have developed guidelines for preprocessing, analyzing, and reporting practices. For the planning stage of fNIRS studies, similar guidance would be desirable. Study preregistration helps researchers to transparently document study protocols before conducting the study - including materials, methods and analyses - and thus others to verify, understand, and reproduce a study. Preregistration can thus serve as a useful tool for transparent, careful and comprehensive fNIRS study design. Aim: We aim to create a guide about the design and analysis steps involved in fNIRS studies and provide a preregistration template specified for fNIRS studies.Approach: The presented preregistration guide has a strong focus on fNIRS specific requirements, and the associated template provides examples based on continuous-wave fNIRS studies conducted in humans. These can, however, be extended to other types of fNIRS studies. Results: On a step-by-step basis, we walk the fNIRS user through key methodological and analysis-related aspects central to a comprehensive fNIRS study design. These include items specific to the design of continuous-wave, task-based fNIRS studies, but also sections that are of general importance, including an in-depth elaboration on sample size planning. Conclusions: Our guide introduces these open science tools to the fNIRS community, providing researchers with an overview of key design aspects and specification recommendations for comprehensive study planning. As such it can be used as a template to preregister fNIRS studies or merely as a tool for transparent fNIRS study design.

show abstract

Reducing bias, increasing transparency, and calibrating confidence with preregistration

Cited by 9 publications

References 150 publications

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Open Science Perspectives on Machine Learning for the Identification of Careless Responding: A New Hope or Phantom Menace?

Using preregistration as a tool for transparent fNIRS study design

Contact Info

Product

Resources

About