The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

Artner, Richard; Verliefde, Thomas; Steegen, Sara; Gomes, Sara; Traets, Frits; Tuerlinckx, Francis; Vanpaemel, Wolf

doi:10.1037/met0000365

Cited by 41 publications

(46 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Studies on computational reproducibility (e.g. [1][2][3][4][5][6]) have found reproducing findings in papers is far from simple. Obstacles include lack of methods descriptions and no availability of source code or even data.…”

Section: Introductionmentioning

confidence: 86%

“…Researchers can choose from a multitude of analysis strategies and if they are not sufficiently described, the likelihood of being able to reproduce the results are low [7,8]. Even in cases where results can be reproduced, it is often tedious and time-consuming to do so [6].…”

Section: Introductionmentioning

confidence: 99%

“…The reason is, that the papers analysed here use very different models, so it is hard to compare them on a single scale (such as absolute relative deviation, see e.g. [6]). We argue, that in combination with a qualitative description of challenges and difficulties that we faced in each reproduction process, this definition fits our small scale, heterogeneous, setting better.…”

Section: Introductionmentioning

confidence: 99%

“…Prepare data for analysis. The description about all these steps was generally vague (see classification of reported results in [6]) meaning that there were multiple ways of preparing or analysing the data that were in line with the descriptions in the original paper. This study, thus, exposed a large amount of "researcher degrees of freedom" [26] coupled with a lack in transparency about in the original studies.…”

mentioning

confidence: 99%

See 3 more Smart Citations

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

et al. 2021

View full text Add to dashboard Cite

Computational reproducibility is a corner stone for sound and credible research. Especially in complex statistical analyses—such as the analysis of longitudinal data—reproducing results is far from simple, especially if no source code is available. In this work we aimed to reproduce analyses of longitudinal data of 11 articles published in PLOS ONE. Inclusion criteria were the availability of data and author consent. We investigated the types of methods and software used and whether we were able to reproduce the data analysis using open source software. Most articles provided overview tables and simple visualisations. Generalised Estimating Equations (GEEs) were the most popular statistical models among the selected articles. Only one article used open source software and only one published part of the analysis code. Replication was difficult in most cases and required reverse engineering of results or contacting the authors. For three articles we were not able to reproduce the results, for another two only parts of them. For all but two articles we had to contact the authors to be able to reproduce the results. Our main learning is that reproducing papers is difficult if no code is supplied and leads to a high burden for those conducting the reproductions. Open data policies in journals are good, but to truly boost reproducibility we suggest adding open code policies.

show abstract

Section: Introductionmentioning

confidence: 86%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Reproducibility is one of the central tenets of credible research practices (Artner et al, 2020;Asendorpf et al, 2013;Bollen et al, 2015;Nelson et al, 2018;Nosek et al, 2021). In the present article, we investigated multiple approaches to synthetic data, which can be used to share surrogate data while protecting the identities of the study participants.…”

Section: Discussionmentioning

confidence: 99%

Using synthetic data to improve the reproducibility of statistical results in psychological research

Grund¹,

Lüdtke²,

Robitzsch³

2021

Preprint

View full text Add to dashboard Cite

In recent years, psychological research has faced a credibility crisis, and open data are often regarded as an important step toward a more reproducible psychological science. However, privacy concerns are among the main reasons that prevent data sharing. Synthetic data procedures, which are based on the multiple imputation (MI) approach to missing data, can be used to replace sensitive data with simulated values, which can be analyzed in place of the original data. One crucial requirement of this approach is that the synthesis model is correctly specified. In this article, we investigated the statistical properties of synthetic data with a particular emphasis on the reproducibility of statistical results. To this end, we compared conventional approaches to synthetic data based on MI with a data-augmented approach (DA-MI) that attempts to combine the advantages of masking methods and synthetic data, thus making the procedure more robust to misspecification. In multiple simulation studies, we found that the good properties of the MI approach strongly depend on the correct specification of the synthesis model, whereas the DA-MI approach can provide useful results even under various types of misspecification. This suggests that the DA-MI approach to synthetic data can provide an important tool that can be used to facilitate data sharing and improve reproducibility in psychological research. In a worked example, we also demonstrate the implementation of these approaches in widely available software, and we provide recommendations for practice.

show abstract

Verification report: Egalitarianism in young children

Clift,

Beaudry,

Leung

et al. 2024

Infant and Child Development

View full text Add to dashboard Cite

The present study sought to evaluate the reproducibility of prominent findings stated by Fehr et al. in their developmental resource allocation experiment "Egalitarianism in Young Children", published in 2008. The experiment involved children making decisions about distributing sweets between themselves and either an in‐group or an out‐group recipient. Fehr et al. found that (1) inequity aversion develops with age; (2) 3‐ to 4‐year‐old children are inclined toward self‐advantageous allocations, whereas 7‐ to 8‐year‐olds distribute sweets more evenly in divisions, and (3) the influence of group status increases as children age. The original article stated that 229 Swiss school students aged 3 to 8 years (102 boys, 127 girls) participated in the study. However, no further demographics were reported. In our attempts to reproduce Fehr et al.'s original analyses and reanalyse the raw dataset, we found that one of the key variables was miscoded. After rectifying the miscoded variable, the reproduction results revealed only one ambiguously irreproducible result regarding a group status main effect in the sharing mini‐game—with three other tests exhibiting either strong reproducibility or ambiguous reproducibility. Reanalysis results indicated that Fehr et al.'s conclusions are robust when tested with alternative analytical tests.Highlights We evaluated the reproducibility of Fehr et al.'s (2008) highly cited developmental resource allocation findings. After identifying an inversely coded variable, we found all tests—bar one probit regression not related to major findings—to be reproducible. Reanalysis confirmed the robustness of Fehr et al.'s conclusions when using alternate analytical methods. Fehr et al.'s developmental resource allocation findings are reproducible and have theoretical implications for understanding inequity aversion and group status effects on resource allocation decisions in young children.

show abstract

The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

Cited by 41 publications

References 60 publications

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

Using synthetic data to improve the reproducibility of statistical results in psychological research

Verification report: Egalitarianism in young children

Contact Info

Product

Resources

About