Examining the Reproducibility of Meta-Analyses in Psychology: A Preliminary Report

Abstract

Meta-analyses are an important tool to evaluate the literature. It is essential that meta-analyses can easily be reproduced to allow researchers to evaluate the impact of subjective choices on meta-analytic effect sizes, but also to update meta-analyses as new data comes in, or as novel statistical techniques (for example to correct for publication bias) are developed. Research in medicine has revealed meta-analyses often cannot be reproduced. In this project, we examined the reproducibility of meta-analyses in psychology by reproducing twenty published meta-analyses. Reproducing published meta-analyses was surprisingly difficult. 96% of meta-analyses published in 2013-2014 did not adhere to reporting guidelines. A third of these meta-analyses did not contain a table specifying all individual effect sizes. Five of the 20 randomly selected meta-analyses we attempted to reproduce could not be reproduced at all due to lack of access to raw data, no details about the effect sizes extracted from each study, or a lack of information about how effect sizes were coded. In the remaining meta-analyses, differences between the reported and reproduced effect size or sample size were common. We discuss a range of possible improvements, such as more clearly indicating which data were used to calculate an effect size, specifying all individual effect sizes, adding detailed information about equations that are used, and how multiple effect size estimates from the same study are combined, but also sharing raw data retrieved from original authors, or unpublished research reports. This project clearly illustrates there is a lot of room for improvement when it comes to the transparency and reproducibility of published meta-analyses.

Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence

“…Despite the potential benefits of meta-analyses, they also leave room for imprecision and subjectivity on part of the researcher (Bangert-Drowns, 1997; Fava, 2002; Lakens, Hilgard, & Staaks, 2016). Although the reproducibility of a single meta-analysis has rarely been examined, available evidence suggests that many meta-analytic results in psychology cannot be reproduced in independent replications (Lakens et al, 2017) and methodological errors are common (Gøtzsche, Hróbjartsson, Marić, & Tendal, 2007). Even less is known about the commonality of meta-analytic results conducted with a similar goal by different research labs.…”

Cumulative Science via Bayesian Posterior Passing

“…This is also true of the stereotype threat literature, which includes seven meta-analyses (Walton & Cohen 2003;Nguyen & Ryan 2008;Walton & Spencer 2009;Stoet & Geary 2012;Picho, Rodriguez & Finnie 2013;Flore & Wicherts 2014;Doyle & Voyer 2016). However, in many cases meta-analyses do not lead to increased certainty (Ferguson 2014;Lakens et al, 2017;Lakens, Hilgard, & Staaks, 2016). One reason is that meta-analyses are not just used to determine whether an effect truly exists but can also be used to reveal the underlying causes of variation across studies.…”

Reproducibility of Individual Effect Sizes in Meta-Analyses in Psychology

“…However, results from Lakens et al (2017), who checked meta-analyses from 2013-2014, show similar results and emphasize a lack of adherence to reporting standards. More research to adherence of reporting standards and subsequent reproducibility of results in recent meta-analyses is needed to help us understand whether there is any sign of improvement.…”

“…However, the numerous choices and judgment calls metaanalysts need to make do not always influence meta-analytic effect size estimates (Aguinis, Dalton, Bosco, Pierce, & Dalton, 2010). Finally, a severe scarcity of relevant information related to effect size extraction, coding, and adhering to reporting guidelines hinders and sometimes obstructs reproducibility in psychological meta-analyses (Lakens et al, 2017).…”