The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines

Abstract

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

“…Obstacles include lack of methods descriptions and no availability of source code or even data. Researchers can choose from a multitude of analysis strategies and if they are not sufficiently described, the likelihood of being able to reproduce the results are low [7,8]. Even in cases where results can be reproduced, it is often tedious and time-consuming to do so [6].…”

Why Ordinal Variables Can (Almost) Always Be Treated as Continuous Variables: Clarifying Assumptions of Robust Continuous and Ordinal Factor Analysis Estimation Methods

“…Finally, we would like to emphasize that modeling ordinal variables with Pearson correlations as well as polychoric correlations can provide complementary information. Moreover, the results of alternative model specifications are worth to be reported (see also Steegen et al, 2016;Hoffmann et al, 2020).…”

The Reliability of Replications: A Study in Computational Reproductions

“…Only three replications may sound promising, but consider that replications are quite rare and appear mostly in isolation. Simply having more single replications will not likely solve the replication crisis, even if we as scientists expected that every study had at least one replicator (Hoffmann et al 2020;Loken and Gelman 2017). When coupled with the fact that many journals tend to avoid replications, especially ones that overturn their own previously published results, and that replications are time-consuming and often not institutionally supported these results might further reduce the appetite for replications among sociologists (Breznau 2021).…”