Adjusting for publication bias in the presence of heterogeneity

Abstract

It is known that the existence of publication bias can influence the conclusions of a meta-analysis. Some methods have been developed to deal with publication bias, but issues remain. One particular method called 'trim and fill' is designed to adjust for publication bias. The method, which is intuitively appealing and comprehensible by non-statisticians, is based on a simple and popular graphical tool called the funnel plot. We present a simulation study designed to evaluate the behaviour of this method. Our results indicate that when the studies are heterogeneous (that is, when they estimate different effects), trim and fill may inappropriately adjust for publication bias where none exists. We found that trim and fill may spuriously adjust for non-existent bias if (i) the variability among studies causes some precisely estimated studies to have effects far from the global mean or (ii) an inverse relationship between treatment efficacy and sample size is introduced by the studies' a priori power calculations. The results suggest that the funnel plot itself is inappropriate for heterogeneous meta-analyses. Selection modelling is an alternative method warranting further study. It performed better than trim and fill in our simulations, although its frequency of convergence varied, depending on the simulation parameters.

Long‐Term Risk of Myocardial Infarction Compared to Recurrent Stroke After Transient Ischemic Attack and Ischemic Stroke: Systematic Review and Meta‐Analysis

“…For instance, looking among all studies, the risk of MI was 1.67%/y, with 95% CI interval at least equal to 1.36%/y and as high as 1.98%/y. As suggested for meta‐analyses in the presence of heterogeneity within studies, the use of a funnel plot and test for small‐effect study is not appropriate 24, 25, 26, 27. Heterogeneity can cause asymmetric funnel plots, even in the absence of publication bias, and may therefore lead to false‐positive claims of publication bias.…”

Publication Bias in the Organizational Sciences

“…However, the results may be inaccurate due to moderating influences (i.e., between-sample heterogeneity). This is particularly likely for our results from funnel plot-based publication bias methods (Sterne et al, , 2011Terrin et al, 2003). Accordingly, we assessed the presence of conceptually identified moderators using meta-regression.…”

“…When this assumption is violated (e.g., sex moderates the magnitude of an effect size), the trim and fill method, as any funnel plotbased method to assess publication bias (e.g., Begg & Mazumdar, 1994;Egger, Smith, Schneider, & Minder, 1997;Peters et al, 2008;, may yield incorrect results (Duval, 2005;Terrin, Schmid, Lau, & Olkin, 2003). Then, trim and fill (and any other funnel plot-based method) can be performed in more homogeneous (i.e., moderator controlled) subgroups of the overall effect size distribution.…”

Can illness beliefs, from the common-sense model, prospectively predict adherence to self-management behaviours? A systematic review and meta-analysis

“…The findings following re-estimation of the meta-analysis were broadly similar, albeit more conservative. It should also be noted that the interpretation of funnel plots be approached with caution, as there may be factors other than reporting bias (for example, delayed publication and selective reporting of outcomes or analyses) that may contribute to funnel plot asymmetry (Sterne et al, 2011;Terrin, Schmid, Lau, & Olkin, 2003). For instance, high heterogeneity and poor methodological quality of studies may result in skewed funnel plots: both of these issues were highly relevant to the present study, and have been discussed.…”