Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Abstract

1. Monitoring the impacts of anthropogenic threats and interventions to mitigate these threats is key to understanding how to best conserve biodiversity. Ecologists use many different study designs to monitor such impacts. Simpler designs lacking controls (e.g. Before-After (BA) and After) or pre-impact data (e.g. Control-Impact (CI)) are considered to be less robust than more complex designs (e.g. Before-After Control-Impact (BACI) or Randomized Controlled Trials (RCTs)). However, we lack quantitative estimates of how much less accurate simpler study designs are in ecology. Understanding this could help prioritize research and weight studies by their design's accuracy in meta-analysis and evidence assessment.2. We compared how accurately five study designs estimated the true effect of a simulated environmental impact that caused a step-change response in a population's density. We derived empirical estimates of several simulation parameters from 47 ecological datasets to ensure our simulations were realistic. We measured design performance by determining the percentage of simulations where: (a) the true effect fell within the 95% Confidence Intervals of effect size estimates, and (b) each design correctly estimated the true effect's direction and magnitude. We also considered how sample size affected their performance.3. We demonstrated that BACI designs performed: 1.3-1.8 times better than RCTs; 2.9-4.2 times versus BA; 3.2-4.6 times versus CI; and 7.1-10.1 times versus After designs (depending on sample size), when correctly estimating true effect's direction and magnitude to within ±30%. Although BACI designs suffered from low power at small sample sizes, they outperformed other designs for almost all performance measures. Increasing sample size improved BACI design accuracy, but only increased the precision of simpler designs around biased estimates. Synthesis and applications.We suggest that more investment in more robust designs is needed in ecology since inferences from simpler designs, even with large sample sizes may be misleading. Facilitating this requires longer-term funding and stronger research-practice partnerships. We also propose 'accuracy weights' and demonstrate how they can weight studies in three recent meta-analyses by | 2743

El Niño impacts on human‐modified tropical forests: Consequences for dung beetle diversity and associated ecological processes

“…Although assessments of the vegetation suggest minimal differences between 2010 and pre‐El Niño samples in 2016, at least some of the influence of El Niño on invertebrate communities may be obscured by pre‐El‐Niño changes in biodiversity through processes such as succession (e.g., Lennox et al, ), longer‐term disturbance responses (e.g., Silva et al, ), or ecological drift and competition (e.g., Levi et al, ; Ulrich, Puchałka, Koprowski, Strona, & Gotelli, ). As such, while before–after studies hold many advantages when the before assessment is immediately before the disturbance (e.g., Christie et al, ; França, Louzada, et al, ), there is a risk that the ecological signal will become degraded with greater temporal disconnection.…”

“…Where large‐scale studies assessing the ecological consequences of wildfires have taken place, they often lack pre‐fire information (e.g., Barlow & Peres, ) and rely on space‐for‐time approaches that may underestimate biotic changes in tropical forests (Christie et al, ; França, Louzada, et al, ). Furthermore, it is not clear how previous anthropogenic forest disturbance, such as selective logging, influences the response of biodiversity and associated functions to ENSO‐mediated droughts and wildfires, or whether changes in biodiversity result in further changes in ecosystems functioning.…”

The challenge of heterogeneous evidence in conservation

“…BA, CI, After). This suggests that we do not only lack studies outside of North America and Western Europe, but also that the few studies that do exist outside these regions are likely to be of poor methodological quality and potentially biased (Christie et al 2019). We should therefore prioritize future research effort on testing conservation interventions using robust study designs in these underrepresented regions and ensuring that they are published.…”

“…Differences in study quality, for example due to the usage of different study designs, may also make it more difficult to decide which studies to trust over others, particularly if they give conflicting results. Several different study designs are used to assess impacts of threats and interventions in ecology (De Palma et al 2018;Christie et al 2019), all of which are affected by different sources and levels of bias and noise. These range from relatively robust designs such as experimental Randomized Controlled Trials (RCTs) and quasi-experimental Before-After Control-Impact designs (BACI), to less robust designs such as Control-Impact (CI), Before-After (BA) and After (also called time series).…”

“…Evidence may also come in the form of systematic reviews and metaanalyses, generally considered robust depending on their methodology and the robustness of the studies they include. Typically, the evidence base for conservation is considered to have relatively few studies with robust study designs, due to logistical, funding, and time-based constraints (De Palma et al 2018;Christie et al 2019). However, we do not know how this broad pattern varies geographically, or how this translates into the prevalence of different designs at an intervention scale.…”