How much does the typical ecological meta‐analysis overestimate the true mean effect size?

Abstract

Many primary research studies in ecology are underpowered, providing very imprecise estimates of effect size. Meta‐analyses partially mitigate this imprecision by combining data from different studies. But meta‐analytic estimates of mean effect size may still remain imprecise, particularly if the meta‐analysis includes a small number of studies. Imprecise, large‐magnitude estimates of mean effect size from small meta‐analyses likely would shrink if additional studies were conducted (regression towards the mean). Here, I propose a way to estimate and correct this regression to the mean, using meta‐meta‐analysis (meta‐analysis of meta‐analyses). Hierarchical random effects meta‐meta‐analysis shrinks estimated mean effect sizes from different meta‐analyses towards the grand mean, bringing those estimated means closer on average to their unknown true values. The intuition is that, if a meta‐analysis reports a mean effect size much larger in magnitude than that reported by other meta‐analyses, that large mean effect size likely is an overestimate. This intuition holds even if different meta‐analyses of different topics have different true mean effect sizes. Drawing on a compilation of data from hundreds of ecological meta‐analyses, I find that the typical (median) ecological meta‐analysis overestimates the absolute magnitude of the true mean effect size by ~10%. Some small ecological meta‐analyses overestimate the magnitude of the true mean effect size by >50%. Meta‐meta‐analysis is a promising tool for improving the accuracy of meta‐analytic estimates of mean effect size, particularly estimates based on just a few studies.

Effective conservation of subterranean‐roosting bats

“…Our meta‐analysis, being strongly focused on the peer‐reviewed literature and to a lesser extent gray literature, may have partly underestimated the total volume of work that is being done for cave bat conservation. The overall low sample size of our meta‐analysis (see Fox [2022] for comparative data) is symptomatic of this, possibly reflecting how the volume of relevant papers is dwarfed by the low numbers of researchers working in conservation. There is a need for researchers and biologists alike to work with unpublished small data sets and compile them into a discoverable source.…”

Impacts of Climate Change and Agricultural Practices on Nitrogen Processes, Genes, and Soil Nitrous Oxide Emissions: A Quantitative Review of Meta-Analyses

“…Meta-analysis has been widely applied in ecological studies to quantify the effects of climate change and agricultural practices on soil N 2 O emissions [30,40]. However, the proliferation of meta-analyses has led to inconsistent findings and occasionally opposing conclusions, necessitating a critical evaluation and synthesis of existing meta-analyses [25,[54][55][56].…”

“…They further applied the approach of mega-analysis, estimating an overall reduction of 38% by synthesizing data from these 18 meta-analyses. Mega-analysis, or a meta-analysis of meta-analyses, offers a more precise estimate of the mean effect size by combining data from various meta-analyses [25,54]. This approach has been found to provide effect size quantification with low bias and high precision [55].…”

Effects of Different Management Measures on Carbon Stocks and Soil Carbon Stocks in Moso Bamboo Forests: Meta-Analysis and Control Experiment

“…Meta-analysis is an analytical tool to integrate and compare multiple studies, and it is also a systematic evaluation method. This approach involves quantifying the results of numerous independent studies by statistically merging them into a single effect size or effect scale to comprehensively reflect the results of these independent studies [21]. The synthesis of responses from these independent studies enhances the credibility of this study and addresses the issue of inconsistent results obtained from individual studies [22,23].…”