Methods for testing publication bias in ecological and evolutionary meta‐analyses

Abstract

1. Publication bias threatens the validity of quantitative evidence from meta-analyses as it results in some findings being overrepresented in meta-analytic datasets because they are published more frequently or sooner (e.g., 'positive' results). Unfortunately, methods to test for the presence of publication bias, or assess its impact on meta-analytic results, are unsuitable for datasets with high heterogeneity and non-independence, as is common in ecology and evolutionary biology.2. We first review both classic and emerging publication bias tests (e.g., funnel plots, Egger's regression, cumulative meta-analysis, fail-safe N, trim-and-fill tests, p-curve and selection models), showing that some tests cannot handle heterogeneity, and, more importantly, none of the methods can deal with non-independence. For each method we estimate current usage in ecology and evolutionary biology, based on a representative sample of 102 meta-analyses published in the last ten years.3. Then, we propose a new method using multilevel meta-regression, which can model both heterogeneity and non-independence, by extending existing regression-based methods (i.e.Egger's regression). We describe how our multilevel meta-regression can test not only publication bias, but also time-lag bias, and how it can be supplemented by residual funnel plots.4. Overall, we provide ecologists and evolutionary biologists with practical recommendations on which methods are appropriate to employ given independent and non-independent effect sizes.No method is ideal, and more simulation studies are required to understand how Type 1 and 2 error rates are impacted by complex data structures. Still, limitations of these methods do not justify ignoring publication bias in ecological and evolutionary meta-analyses.

A meta‐analysis of sex differences in animal personality: no evidence for the greater male variability hypothesis

“…We therefore included the inverse square root of 'effective sample size' (1/ñ i ) as a moderator term in our MLMR models to test whether it explained some of the variation in the reported effect sizes (for methodological justification see Nakagawa et al, 2021b). If the inverse of effective sample size has a significant influence, this is suggestive of either unbalanced sampling or publication bias (Nakagawa et al, 2021b). The inverse of ñ i is calculated as (female n + male n)/ (female n × male n) (see Nakagawa et al, 2021b).…”

Physiology can predict animal activity, exploration, and dispersal

“…] as covariates to account for time-lag and publication bias 82 . We calculated an extended heterogeneity (I²) statistic to partition total heterogeneity (I² total) into within-species variance (I² species), phylogenetic variance (I² phylogeny), study ID variance (I² study) and residual variance (I² effect size) 83 .…”

“…All models included four random factors as above (except the taxon model), with publication year and the square root of the inverse of effective sample size ( ffiffiffiffiffiffiffiffiffi 1=e n i p ) as covariate terms, which account for time-lag and publication bias. The effective sample size was preferable here because it accounts for unbalanced sampling between groups 82 . The inverse of effective sample size (1=e n i ; Eq.…”

Meta-analytic approaches and effect sizes to account for ‘nuisance heterogeneity’ in comparative physiology

“…Second, there are sociological factors impacting heterogeneity. These include the ease with which novel and significant results are published relative to non-significant results, manifesting as publication bias (Jennions et al, 2013;Nakagawa et al, 2021a;Rothstein et al, 2005). Third, real biological processes can drive effect heterogeneity.…”

“…Let us assume M 1 is a random variable drawn from a distribution that can be characterised by a mean and standard deviation (note that this standard deviation is not the 'sample' but the 'sampling' standard deviation, which is often referred to as standard error; see fig. 1 in Nakagawa et al, 2021a). Multiplying it by a constant (a) will change the variance by the square of that constant (a 2 ) while adding or subtracting the constant (b) does not change the variance of M 1 .…”