Preprint - Meta-Analyzing the Multiverse: A Peek Under the Hood of Selective Reporting

Olsson-Collentine, Anton; Rcm, van Aert; Bakker, Marjan; Wicherts, Jelte M.

doi:10.31234/osf.io/43yae

Cited by 5 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All our code and data for this project are available on the Open Science Framework (OSF) at http://osf.io/j8yg2/ (Olsson-Collentine et al, 2019), and permanently archived at Zenodo at doi.org/10 .5281/zenodo.7341292 (Olsson-Collentine et al, 2022). We refer directly to relevant files on the OSF using brackets and links in the sections below.…”

Section: Transparency and Opennessmentioning

confidence: 99%

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Olsson-Collentine¹,

Aert²,

Bakker³

et al. 2023

Psychological Methods

View full text Add to dashboard Cite

Researcher degrees of freedom refer to arbitrary decisions in the execution and reporting of hypothesis-testing research that allow for many possible outcomes from a single study. Selective reporting of results ( p-hacking) from this "multiverse" of outcomes can inflate effect size estimates and false positive rates. We studied the effects of researcher degrees of freedom and selective reporting using empirical data from extensive multistudy projects in psychology (Registered Replication Reports) featuring 211 samples and 14 dependent variables. We used a counterfactual design to examine what biases could have emerged if the studies (and ensuing meta-analyses) had not been preregistered and could have been subjected to selective reporting based on the significance of the outcomes in the primary studies. Our results show the substantial variability in effect sizes that researcher degrees of freedom can create in relatively standard psychological studies, and how selective reporting of outcomes can alter conclusions and introduce bias in meta-analysis. Despite the typically thousands of outcomes appearing in the multiverses of the 294 included studies, only in about 30% of studies did significant effect sizes in the hypothesized direction emerge. We also observed that the effect of a particular researcher degree of freedom was inconsistent across replication studies using the same protocol, meaning multiverse analyses often fail to replicate across samples. We recommend hypothesis-testing researchers to preregister their preferred analysis and openly report multiverse analysis. We propose a descriptive index (underlying multiverse variability) that quantifies the robustness of results across alternative ways to analyze the data.

show abstract

Section: Transparency and Opennessmentioning

confidence: 99%

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Olsson-Collentine¹,

Aert²,

Bakker³

et al. 2023

Psychological Methods

View full text Add to dashboard Cite

show abstract

“…At last, one could argue that the results of factor analyses are dependent upon many parameters that are chosen by the researcher (see multiverse debate Olsson-Collentine et al 2020;Steegen et al 2016), which questions the replicability of the findings. To address this issue, we ran multiple factor analyses with different indexes combinations and different parameters, and the results' pattern remained mainly unchanged.…”

Section: Limitationsmentioning

confidence: 99%

Comparing Self-Other Distinction Across Motor, Cognitive, and Affective Domains

Pronizius¹,

Bukowski²,

Lamm³

2022

Preprint

View full text Add to dashboard Cite

Self-other distinction (SOD) is a process, with which we disentangle self- from other-related mental representations. This online study investigated two unresolved questions: (1) whether SOD is underpinned by a unitary process across three types of mentalrepresentations: motor, cognitive, and affective representations, and (2) whether SOD largelycoincides with either domain-general processes related to cognitive control or social cognition-specific processes. Participants performed three well-established SOD tasks (motor representations: automatic imitation inhibition; cognitive representations: visual perspective-taking; affective representations: emotional egocentricity bias) and two cognitive control tasks (Stroop and stop-signal reaction time tasks).Regarding question 1, multiple exploratory factor analyses on the SOD indexes across three types of mental representations resulted in a three-factor solution (30% of explained variance) instead of a single factor on which all SOD indexes could have converged. Regarding question 2, adding cognitive control indexes to the factor analyses did not alter the previous three-factorsolution and the control indexes had relatively low loadings (below .30) on the three identifiedfactors. These findings suggest that (1) SOD for motor, cognitive, and affective mental representations is supported by distinct and not a unitary process and (2) that SODs recruit domain-general processes related to a cognitive control only to a limited extent.

show abstract

“…This report might be, for example a p -value distribution, a specification curve analysis and plot, or a vibration of effects plot (for review, see Hall et al, 2022). A joint conclusion can then be drawn by integrating the results across the multiverse of options (Simonsohn et al, 2020) or by reporting a multiverse variability index that quantifies the robustness of the results across the forking paths (Olsson-collentine et al, 2023).…”

Section: Introductionmentioning

confidence: 99%

The multiverse of data preprocessing and analysis in graph-based fMRI: A systematic literature review of analytical choices fed into a decision support tool for informed analysis

Kristanto,

Burkhardt,

Thiel

et al. 2024

Preprint

View full text Add to dashboard Cite

The large number of different analytical choices researchers use may be partly responsible for the replication challenge in neuroimaging studies. For robustness analysis, knowledge of the full space of options is essential. We conducted a systematic literature review to identify the analytical decisions in functional neuroimaging data preprocessing and analysis in the emerging field of cognitive network neuroscience. We found 61 different steps, with 17 of them having debatable options. Scrubbing, global signal regression, and spatial smoothing are among the controversial steps. There is no standardized order in which different steps are applied, and the options within several steps vary widely across studies. By aggregating the pipelines across studies, we propose three taxonomic levels to categorize analytical choices: 1) inclusion or exclusion of specific steps, 2) distinct sequencing of steps, and 3) parameter tuning within steps. To facilitate access to the data, we developed a decision support app with high educational value called METEOR, which allows researchers to explore the space of choices as reference for well-informed robustness (multiverse) analysis.

show abstract

Preprint - Meta-Analyzing the Multiverse: A Peek Under the Hood of Selective Reporting

Cited by 5 publications

References 21 publications

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Comparing Self-Other Distinction Across Motor, Cognitive, and Affective Domains

The multiverse of data preprocessing and analysis in graph-based fMRI: A systematic literature review of analytical choices fed into a decision support tool for informed analysis

Contact Info

Product

Resources

About