The Effect of Publication Bias on the Assessment of Heterogeneity

Abstract

One of the main goals of meta-analysis is to test and estimate the heterogeneity of effect size. We examined the effect of publication bias on the Q-test and assessments of heterogeneity, as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The expected values of heterogeneity measures H2 and I2 were analytically derived, and the power and the type I error rate of the Q-test were examined in a Monte-Carlo simulation study. Our results show that the effect of publication bias on the Q-test and assessment of heterogeneity is large, complex, and non-linear. Publication bias can both dramatically decrease and increase heterogeneity. Extreme homogeneity can occur even when the population heterogeneity is large. Particularly if the number of studies is large and population effect size is small, publication bias can cause both extreme type I error rates and power of the Q-test close to 0 or 1. We therefore conclude that the Q-test of homogeneity and heterogeneity measures H2 and I2 are generally not valid in assessing and testing heterogeneity when publication bias is present, especially when the true effect size is small and the number of studies is large. We introduce a web application, Q-sense, which can be used to assess the sensitivity of the Q-test to publication bias, and we apply it to two published meta-analysis. Meta-analytic methods should be enhanced in order to be able to deal with publication bias in their assessment and tests of heterogeneity.

Neglect of publication bias compromises meta-analyses of educational research

“…Instead of factoring out study idiosyncrasies and protecting the consumers of research from single-study biases, an unadjusted meta-analysis (under publication bias) tends to find a significant overall effect regardless of whether it is there or not. If there is a publication bias, including more NEGLECT OF PUBLICATION BIAS IN EDUCATIONAL RESEARCH published studies only exacerbates the problem (Augusteijn et al, 2019). Evidence synthesis of biased literature, therefore, does not represent a severe test, making fallacious theories hardly falsifiable (see Mayo, 2018).…”

Strengths and Weaknesses of Meta-Analyses

“…van Aert and colleagues even went so far as to say "In case of strong evidence or strong indications of p-hacking, [metaanalysts should] be reluctant in interpreting estimates of traditional meta-analytic techniques and p-uniform and p-curve because their effect-size estimates may be biased in any direction depending on the type of p-hacking used" (p.714). It is also currently unknown how QRPs affect heterogeneity estimates (Augusteijn et al, 2019). Carter et al's (2019) simulations suggest that the effects of publication bias may be somewhat larger than the effects of QRPs under commonly observed conditions in the social sciences, although the generalizability of this conclusion might depend on how closely the conditions of their simulations do match the real world, as well as which type and combination of QRPs are used.…”

“…Further, the weight function, even if estimated, may not be well recovered by the model (Hedges & Vevea, 2005). Crucially, the performance of the model depends on the accuracy of the selection function, which is to some extent unknowable (Augusteijn et al, 2019). Carter et al (2019) report that the 3PSM showed reasonable type I error rates but was underpowered under many simulated conditions that match those often observed in the social sciences.…”

“…Carter et al (2019) offer a web application, metaExplorer, which researchers can use to assess when a particular technique may or may not be appropriate for their review. Augusteijn et al (2019) also offer Qsense, a web application that can help researchers understand how heterogeneity estimates may be biased under varying conditions.…”

“…Publication bias can also affect heterogeneity estimates in somewhat unpredictable ways (seeAugusteijn et al, 2019). These authors summarize results stating: "the Q-test of homogeneity and heterogeneity measures H 2 and I 2 are generally not valid when publication bias is present" (p. 1).…”