Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology

Abstract

Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%-38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%-12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2-3 times (Type M errors) and variability by 4-10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts.Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experimentsThis is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

Are experiment sample sizes adequate to detect biologically important interactions between multiple stressors?

“…In our examples, the mean detected interaction effect size only approaches the true interaction effect size at around n = 40, and at small sample sizes the mean detected effect size is approximately three times the magnitude of the true effect size (Figure 5b ). This shows how publishing only statistically significant results from experiments with low sample sizes leads to an overestimation of nonadditivity, a problem that has also been highlighted for biological responses to single stressors (Yang et al, 2022 ).…”

“…Commonplace sample sizes (e.g., 4 replicates per treatment) are not adequate for this question (Figures 4 and 5 ), and the researcher will likely need to implement sample sizes that are multiple (two or more) times larger than those commonly used. There may be other situations where a smaller effect is not so important, implying smaller samples are adequate, such as monitoring abundance declines in a system with high functional redundancy, but even here, care needs to be taken since concerns have been raised regarding publication bias leading to the overestimation of stressor effects from experiments with small sample sizes (Figure 5b , Yang et al, 2022 ).…”

“…Unfortunately, it is much harder to increase the sample sizes of many mesocosm experiments for these higher levels of organization simply due to the financial cost, space, and time required to manage large sample sizes for all four treatments (Boyd et al, 2018 ). One alternative to boost within‐study replication is to use coordinated networks of researchers who ask the same experimental question(s) across multiple sites, using the same protocol (Filazzola & Cahill, 2021 ; Yang et al, 2022 ). An example of this is the Nutrient Network (NutNet) organization ( https://nutnet.org/ ) that among its key questions asks: To what extent are plant production and diversity co‐limited by multiple nutrients in herbaceous‐dominated communities?…”

“…The key point is that to aid general understanding, and avoid publication bias (e.g., Figure 5b ), it is crucial that all experiments are published with the data made openly available (i.e., the three components of sample size, mean, and standard deviation/error or variance for each treatment) and not just those experiments that detect “interesting” non‐null stressor‐pair interactions (Filazzola & Cahill, 2021 ). Indeed, it is likely that publication bias is leading to the effects of anthropogenic stressors being overestimated (Yang et al, 2022 ), while multiple stressor ecology suffers from the erroneous over‐reporting of synergistic interactions (Côté et al, 2016 ). Unfortunately, there are still many papers that do not report or make their data (i.e., treatment means etc.)…”

Untitled

“…This biased publishing can result in the proliferation of studies reporting strong effects, even though they may not be true [26] and can fuel citation bias [27]. Indeed, a recent analysis suggested that field studies in global change biology suffer from publication bias, which has fuelled the proliferation of underpowered studies reporting overestimated effect sizes [28]. To determine if studies testing for effects of ocean acidification on fish behavior exhibited signs of publication bias and citation bias, we assessed relationships between effect size magnitude, journal impact factor, and Google Scholar citations (Fig 4).…”