Does restoration work? It depends on how we measure success

Mahlum, Shad; Côté, David; Wiersma, Yolanda F.; Pennell, C. J.; Adams, Blair

doi:10.1111/rec.12649

Cited by 12 publications

(12 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results provide strong evidence that simpler designs (e.g. After, BA and CI) often yield different inferences to BACI designs, as observed empirically by previous studies (França et al, ; Mahlum et al, ; Osenberg et al, ; Smokorowski & Randall, ). We also found that BA and CI designs were more prone to underestimation than overestimation (Figure b), which is consistent with results from França et al () that showed a CI design underestimated the impacts of logging relative to a BACI design.…”

Section: Discussionsupporting

confidence: 84%

“…The disparities between the robustness of study designs and their usage is concerning as many studies may be making misleading inferences about anthropogenic impacts. Some empirical comparisons of the consequences of using BACI, BA and CI designs have been undertaken (Osenberg et al, ; França et al, ; Mahlum et al, ; Smokorowski & Randall, ). However, we are yet to understand how inaccurate simpler designs are relative to complex ones, or the influence of sample size on these patterns (e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Christie

Amano

Martin

et al. 2019

Journal of Applied Ecology

194

254

View full text Add to dashboard Cite

1. Monitoring the impacts of anthropogenic threats and interventions to mitigate these threats is key to understanding how to best conserve biodiversity. Ecologists use many different study designs to monitor such impacts. Simpler designs lacking controls (e.g. Before-After (BA) and After) or pre-impact data (e.g. Control-Impact (CI)) are considered to be less robust than more complex designs (e.g. Before-After Control-Impact (BACI) or Randomized Controlled Trials (RCTs)). However, we lack quantitative estimates of how much less accurate simpler study designs are in ecology. Understanding this could help prioritize research and weight studies by their design's accuracy in meta-analysis and evidence assessment.2. We compared how accurately five study designs estimated the true effect of a simulated environmental impact that caused a step-change response in a population's density. We derived empirical estimates of several simulation parameters from 47 ecological datasets to ensure our simulations were realistic. We measured design performance by determining the percentage of simulations where: (a) the true effect fell within the 95% Confidence Intervals of effect size estimates, and (b) each design correctly estimated the true effect's direction and magnitude. We also considered how sample size affected their performance.3. We demonstrated that BACI designs performed: 1.3-1.8 times better than RCTs; 2.9-4.2 times versus BA; 3.2-4.6 times versus CI; and 7.1-10.1 times versus After designs (depending on sample size), when correctly estimating true effect's direction and magnitude to within ±30%. Although BACI designs suffered from low power at small sample sizes, they outperformed other designs for almost all performance measures. Increasing sample size improved BACI design accuracy, but only increased the precision of simpler designs around biased estimates. Synthesis and applications.We suggest that more investment in more robust designs is needed in ecology since inferences from simpler designs, even with large sample sizes may be misleading. Facilitating this requires longer-term funding and stronger research-practice partnerships. We also propose 'accuracy weights' and demonstrate how they can weight studies in three recent meta-analyses by | 2743

show abstract

Section: Discussionsupporting

confidence: 84%

Section: Introductionmentioning

confidence: 99%

Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Christie

Amano

Martin

et al. 2019

Journal of Applied Ecology

194

254

View full text Add to dashboard Cite

show abstract

“…Our results provide strong evidence that simpler study designs (e.g. BA and CI) often yield different effect sizes and inferences to BACI designs, as observed empirically by previous studies (Osenberg et al 2011;França et al 2016;Mahlum 2018). Therefore, studies using After, BA and CI designs seriously risk presenting misleading conclusions on the impact of threats and interventions.…”

Section: Discussionsupporting

confidence: 67%

“…The disparities between the robustness of study designs and their usage is problematic because this may mean we are making misleading inferences about anthropogenic impacts. Although some empirical comparisons of the consequences of using BACI, BA and CI designs have been undertaken (Osenberg et al 2011;França et al 2016;Mahlum 2018), we still do not understand how much less accurate simpler designs are than more complex ones, or the influence on accuracy of increasing sample size of different study designs. A quantitative comparison of the accuracy of different study designs and their sample size would help us better understand these issues.…”

Section: Introductionmentioning

confidence: 99%

Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Christie

Amano

Martin

et al. 2019

Preprint

View full text Add to dashboard Cite

1. Ecologists use a wide range of study designs to estimate the impact of interventions or threats but there are no quantitative comparisons of their accuracy. For example, while it is accepted that simpler designs, such as After (sampling sites post-impact without a control), Before-After (BA) and Control-Impact (CI), are less robust than Randomised Controlled Trials (RCT) and Before-After Control-Impact (BACI) designs, it is not known how much less accurate they are.2. We simulate a step-change response of a population to an environmental impact using empirically-derived estimates of the major parameters. We use five ecological study designs to estimate the effect of this impact and evaluate each one by determining the percentage of simulations in which they accurately estimate the direction and magnitude of the environmental impact. We also simulate different numbers of replicates and assess several accuracy thresholds.3. We demonstrate that BACI designs could be 1.1-1.5 times more accurate than RCTs, 2.9-4.1 times more accurate than BA, 3.8-5.6 times more accurate than CI, and 6.8-10.8 times more accurate than After designs, when estimating to within ±30% of the true effect (depending on the sample size). We also found that increasing sample size substantially increases the accuracy of BACI designs but only increases the precision of simpler designs around a biased estimate; only by using more robust designs can accuracy increase. Modestly increasing replication of both control and impact sites also increased the accuracy of BACI designs more than substantially increasing replicates in just one of these groups.4. We argue that investment into using more robust designs in ecology, where possible, is extremely worthwhile given the inaccuracy of simpler designs, even when using large sample sizes. Based on our results we propose a weighting system that quantitatively ranks the accuracy of studies based on their study design and the number of replicates used. We hope these 'accuracy weights' enable researchers to better account for study design in evidence synthesis when assessing the reliability of a range of studies using a variety of designs.

show abstract

“…An additional general linear model was conducted to analyze the effect of the measures on goby size in the littoral zone, with square root-transformed total lengths of bighead and round goby as response variables and section and survey as factors. Statistical significance of the effects of the respective measures on goby abundance or size are revealed by the interaction term section x survey (Mahlum et al 2018). The level of significance was set at 5%.…”

Section: Discussionmentioning

confidence: 99%

Effects of hydraulic engineering restoration measures on invasive gobies in a large river (Danube, Austria)

Ramler

Keckeis

2019

Biol Invasions

View full text Add to dashboard Cite

Invasive gobies can have severe detrimental effects on local fish communities, however, direct methods for population control are often insufficient (i.e. fishing) or not feasible (i.e. poisoning). Indirect methods, such as habitat modifications in the course of restoration programs, appear promising but are poorly studied. In this study, we investigate the effects of different restoration measures on the abundance and occurrence of non-native gobies in the main stem of a free-flowing section of the Danube and attempt to disentangle these measures from general large-scale trends by applying a Before-After-Control-Impact design. We found three invasive goby species (racer, bighead, and round goby) in the sampling area, partly with very high abundances. Four to six years after the installation, the measures had negative (riprap removal), neutral (side arm reconnection), or positive (groyne field adaptations) effects on goby abundances. We conclude that the impact of the measures depends on the type of intervention, is species-specific, and is largely related to substrate composition. Independent from the effect of the measures, abundances of bighead and round goby dropped in the project and reference sections after the pre-survey. This general decline probably indicates a stabilization phase of the goby populations on a lower level, but may also be influenced by a major flood event. Nevertheless, our results indicate a high potential of shoreline modifications for invasive species control, calling for considering and incorporating them in river restoration programs.

show abstract

Does restoration work? It depends on how we measure success

Cited by 12 publications

References 59 publications

Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Simple study designs in ecology produce inaccurate estimates of biodiversity responses

Effects of hydraulic engineering restoration measures on invasive gobies in a large river (Danube, Austria)

Contact Info

Product

Resources

About