Adjusting for Publication Bias in Meta-Analysis

Abstract

We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysisthat is, exploring the range of estimates that result from assuming different forms of and severity of publication bias.

Gender differences in depression in representative national samples: Meta-analyses of diagnoses and symptoms.

“…A recent review on adjusting for publication bias in meta-analysis encouraged the use of sensitivity measures (McShane, Böckenholt, & Hansen, 2016). The likelihood ratio tests (LRT) comparing the unadjusted to adjusted models (using p -value cut points of 0.05, 0.01, and 0.001) for the major depression and depression symptom meta-analyses, respectively, were not statistically significant, p = 0.073 and p = 0.3226.…”

Does your mindfulness benefit others? A systematic review and meta‐analysis of the link between mindfulness and prosocial behaviour

“…We found no evidence of publication bias for correlational studies, but some evidence among intervention studies. Although several contemporary approaches to assessing publication bias have been proposed, such as p-curve, p-uniform, PET-PEESE, 3PSM (for reviews, see Carter, Sch€ onbrodt, Gervais, & Hilgard, 2018;and McShane, B€ ockenholt, & Hansen, 2016), a key advantage of the moderation tests we undertook is that they do not assume independence among effect sizes, which was a significant issue in the data included in this review. The complete results of the risk of bias and publication bias assessments are reported in Appendix S1.…”

Infant Motor Development in Autism Spectrum Disorder: A Synthesis and Meta‐analysis

“…Finally, a threeparameter selection method was used, which computes a weighted mean effect size with an adjustment for publication bias. This new adjusted effect size is then compared to the original synthesized effect size (McShane, B€ ockenholt, & Hansen, 2016).…”