Computational Reproducibility via Containers in Psychology

Clyburne-Sherin, April; Fei, Xu; Green, Seth Ariel

doi:10.15626/mp.2018.892

Cited by 52 publications

(35 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data from the papers included here were checked by multiple coders, see Bakdash et al (2020c) for details. Our results are reproducible using our materials on the Open Science Framework ( Bakdash et al, 2020d ) or with the Code Ocean platform ( Clyburne-Sherin et al, 2019 ) using our capsule ( Bakdash et al, 2020e ).…”

Section: Methods and Resultssupporting

confidence: 55%

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Bakdash¹,

Marusich²,

Kenworthy³

et al. 2020

Front. Psychol.

View full text Add to dashboard Cite

Whether in meta-analysis or single experiments, selecting results based on statistical significance leads to overestimated effect sizes, impeding falsification. We critique a quantitative synthesis that used significance to score and select previously published effects for situation awareness-performance associations (Endsley, 2019). How much does selection using statistical significance quantitatively impact results in a meta-analytic context? We evaluate and compare results using significance-filtered effects versus analyses with all effects as-reported. Endsley reported high predictiveness scores and large positive mean correlations but used atypical methods: the hypothesis was used to select papers and effects. Papers were assigned the maximum predictiveness scores if they contained at-least-one significant effect, yet most papers reported multiple effects, and the number of non-significant effects did not impact the score. Thus, the predictiveness score was rarely less than the maximum. In addition, only significant effects were included in Endsley’s quantitative synthesis. Filtering excluded half of all reported effects, with guaranteed minimum effect sizes based on sample size. Results for filtered compared to as-reported effects clearly diverged. Compared to the mean of as-reported effects, the filtered mean was overestimated by 56%. Furthermore, 92% (or 222 out of 241) of the as-reported effects were below the mean of filtered effects. We conclude that outcome-dependent selection of effects is circular, predetermining results and running contrary to the purpose of meta-analysis. Instead of using significance to score and filter effects, meta-analyses should follow established research practices.

show abstract

Section: Methods and Resultssupporting

confidence: 55%

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Bakdash¹,

Marusich²,

Kenworthy³

et al. 2020

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…Some fields, such as parts of high energy physics, have a tradition of setting up their full analysis pipelines prior to feeding it any real data, and running tests on the pipeline to check for errors. Software tools such as Docker (Boettiger, 2015;Clyburne-Sherin et al, 2019) allow anyone to archive a fully working version of their analysis code. A field that prioritizes self-correction is one that invests in developing and using these kinds of tools, and that is shown to have few such errors when these tools are applied to its published literature.…”

Section: Critical Appraisal Moscsmentioning

confidence: 99%

Where Are The Self-Correcting Mechanisms In Science?

Vazire¹,

Holcombe²

2020

Preprint

View full text Add to dashboard Cite

It is often said that science is self-correcting, but the replication crisis suggests that, at least in some fields, self-correction mechanisms have fallen short of what we might hope for. How can we know whether a particular scientific field has effective self-correction mechanisms, that is, whether its findings are credible? The usual processes that supposedly provide mechanisms for scientific self-correction – mainly peer review and disciplinary committees – have been inadequate. We argue for more verifiable indicators of a field’s commitment to self-correction. These include transparency, which is already a target of many reform efforts, and critical appraisal, which has received less attention. Only by obtaining Measurements of Observable Self-Correction (MOSCs) can we begin to evaluate the claim that “science is self-correcting.” We expect the validity of this claim to vary across fields and subfields, and suggest that some fields, such as psychology and biomedicine, fall far short of an appropriate level of transparency and, especially, critical appraisal. Fields without robust, verifiable mechanisms for transparency and critical appraisal cannot reasonably be said to be self-correcting, and thus do not warrant the credibility often imputed to science as a whole.

show abstract

“…Even though other containerization platforms exist, such as CodeOcean (described in detail in Clyburne-Sherin et al, 2019) and Singularity (sylabs.io), we chose to use Docker for this tutorial for five main reasons. First, Docker is one of the leading container platforms and has been established as best practice in several research fields (Boettiger, 2015).…”

Section: Brief Introduction To Dockermentioning

confidence: 99%

Leveraging Containers for Reproducible Psychological Research

Wiebels

Moreau

2021

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

Containers have become increasingly popular in computing and software engineering and are gaining traction in scientific research. They allow packaging up all code and dependencies to ensure that analyses run reliably across a range of operating systems and software versions. Despite being a crucial component for reproducible science, containerization has yet to become mainstream in psychology. In this tutorial, we describe the logic behind containers, what they are, and the practical problems they can solve. We walk the reader through the implementation of containerization within a research workflow with examples using Docker and R. Specifically, we describe how to use existing containers, build personalized containers, and share containers alongside publications. We provide a worked example that includes all steps required to set up a container for a research project and can easily be adapted and extended. We conclude with a discussion of the possibilities afforded by the large-scale adoption of containerization, especially in the context of cumulative, open science, toward a more efficient and inclusive research ecosystem.

show abstract

Computational Reproducibility via Containers in Psychology

Cited by 52 publications

References 32 publications

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Where Are The Self-Correcting Mechanisms In Science?

Leveraging Containers for Reproducible Psychological Research

Contact Info

Product

Resources

About