Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences

Abstract

Meta-analysis is a quantitative way of synthesizing results from multiple studies to obtain reliable evidence of an intervention or phenomenon. Indeed, an increasing number of meta-analyses are conducted in environmental sciences, and resulting meta-analytic evidence is often used in environmental policies and decision-making. We conducted a survey of recent meta-analyses in environmental sciences and found poor standards of current meta-analytic practice and reporting. Only ~40% of the 73 reviewed meta-analyses reported heterogeneity (variation among effect sizes beyond sampling error), and publication bias was assessed in less than half. Furthermore, although almost all the meta-analyses had multiple effect sizes originating from the same studies, non-independence among effect sizes was considered in only half of the meta-analyses. To improve the implementation of meta-analysis in environmental sciences, we here outline practical guidance for conducting a meta-analysis in environmental sciences. We describe the key concepts of effect size statistics and meta-analysis, and detail procedures for fitting multilevel meta-analysis and meta-regression models and performing associated publication bias tests. We demonstrate a clear need for environmental scientists to embrace multilevel meta-analytic models, which explicitly model dependence among effect sizes, rather than the commonly used random-effects models. Further, we discuss how reporting and visual presentations of meta-analytic results can be much improved by following reporting guidelines such as PRISMA-EcoEvo (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Ecology and Evolutionary Biology). This paper, along with the accompanying online tutorial (link), serves as a practical guide on conducting a complete set of meta-analytic procedures (i.e., meta-analysis, heterogeneity quantification, meta-regression, publication bias tests and sensitivity analysis) and also as a gateway to more advanced, yet appropriate, methods.

Meta‐analysis reveals that the effects of precipitation change on soil and litter fauna in forests depend on body size

“…We used multilevel meta‐analytical models with inverse variance weighting as implemented in the R package metafor (Viechtbauer, 2010). To examine the impacts of precipitation reduction or increase on soil and litter fauna abundance, taxonomic richness and Shannon–Wiener diversity index (Figure 1a), we built models with no modifiers that included study and site as nested random effects to account for the lack of independence between observations from the same study and site (Nakagawa, Noble, et al., 2023; Nakagawa, Yang, et al., 2023). We chose not to combine the different outcomes for diversity as doing so can blur responses and limit interpretability of results (Liu et al., 2023).…”

Herbivore diversity effects on Arctic tundra ecosystems: a systematic review

“…We expect the effects of multiple herbivores on ecosystem dynamics to be greater than the effects of single groups of herbivores, although the slope and direction of this relationship can change depending on the response being considered Barbero-Palacios et al Environmental Evidence (2024) 13:6 tundra ecosystems. The methods of this review follow a published systematic review protocol [24] and the recently published practical guide for conducting quantitative syntheses in environmental sciences [27]. Full details on literature searches and raw data coded are provided in Additional file 2 and Additional file 3.…”

“…Data availability for these variables differed, so only some of them were considered in the models (see Data synthesis). -Other study information: to assess potential sources of publication bias, we included effective sample size (small study effect) and publication year of the study (decline effect) as recommended by [27]. Overall risk of bias (see Study validity assessment) was also considered as a source of heterogeneity.…”

“…When studies did not report any measurement of variability (448 studies), we imputed missing values using the function "impute_SD" in the package metagear [48] as recommended by [27], applied to each outcome variable. This function fills in missing values of SD based on resampling techniques using the coefficient of variation of available complete records (i.e., studies reporting the same outcome variable that did report SD)…”

ABCal: a Python package for Author Bias Computation and Scientometric Plotting for Reviews and Meta-Analyses

“…This avoids some of the pitfalls and biases that could influence narrative reviews (Pae 2015;Tawfik et al 2019). Another advantage of systematic reviews is the possibility to perform a meta-analysis and scientometric assessment of the included studies (Nakagawa et al 2023). This is done using primary reported statistics to derive the effect size (Cohen 1988) or treatment effect (TE), and variance or standard error (SETE), of the measured outcome.…”