A comparison of common metrics used to quantify the effectiveness of conservation interventions

Abstract

Background Evidence-based conservation is urgently needed to identify, apply and promote effective interventions for mitigation of threats and recovery of the natural environment. Estimation of intervention effectiveness is subject to robust study design and statistical analysis, and much progress is documented in these fields. In contrast, little is understood about the accuracy and biases (underestimation and overestimation) of different effectiveness metrics and how they are affected by sample size. Methods In this study, a dataset (n = 500 cases) consisting of random, positive, integer numbers was simulated to produce frequency input data for the 2 × 2 contingency table. For each case, three metrics of the relative risk, odds ratio and the magnitude of change were calculated, their disparity was estimated and the characteristics of treatment (with intervention) and control (without intervention) samples significantly affecting this disparity were studied by means of linear regression. Further, four case studies from different conservation interventions are provided to support the results. Results The study has shown that the relative risk and the magnitude of change produce identical estimates of intervention effectiveness only when treatment and control samples are equal, or when the number of target outcomes (e.g., number of livestock killed by predators) in treatment sample reaches zero. In other situations, the magnitude of change gives overestimates or underestimates, depending on relationships between treatment and control sample sizes. The table summarizing the conditions of equalities and biases between these two metrics is provided. These conditions are valid for both reduction-aimed interventions reducing negative target outcomes (e.g., livestock protection to reduce livestock losses to predators) and for addition-aimed interventions increasing positive target outcomes (e.g., establishment of protected areas to increase species presence). No significant effects on the odds ratio were found. Conclusion Researchers should set equal treatment and control sample sizes so that to produce identical estimates of intervention effectiveness by the relative risk and the magnitude of change. Otherwise, these estimates are biased if produced by the magnitude of change and the relative risk should be used instead. As setting equal treatment and control samples can be impractical, I encourage researchers and practitioners to use the relative risk in estimation of intervention effectiveness. This will not take additional efforts as both metrics are calculated from the same contingency table. Treatment and control sample sizes, along with their sub-samples affected and not by an intervention, should be explicitly reported by researchers to allow independent evaluation of intervention effectiveness. This approach can help obtain more accurate information on intervention effectiveness for making better decisions in conservation actions.

Perceptions and realities of elephant crop raiding and mitigation methods

“…Surprisingly, none of the three indicators employed (likelihood of farm intrusion, crop damage by elephant, or the extent of monetary losses) provided strong evidence for the effectiveness of either beehive or chili fences in reducing crop damages caused by elephants. In part, nonsignificance of treatment effects may be the result of limited spatial replication and unequal sample sizes of treatments (especially considering few beehive treatments), possibly resulting in low test power (Khorozyan, 2020). Nevertheless, nonsignificant reduction of crop damage (and occasional greater elephant intrusion and damage likelihood in treatments relative to control sites) is surprising because previous studies reported strong evidence for the effectiveness of beehive (King et al, 2011, 2017; Scheijen et al, 2019), and chili (Chang'a et al, 2016; Davies et al, 2011) fences.…”

“…In close cooperation with the 20 farmers (farmers expressed their willingness to support the project and we agreed upon equal benefit sharing of bee products), we distributed two beehive fences and 10 chili fences to the plots; eight sites were left untreated as a control (Figure 1). Ideally, experiments to assess conservation evidence distribute an equal number of treatments and controls (Khorozyan, 2020), yet funding constraints limited us to two beehive treatments only. As we worked in collaboration with the farmers and some farmers were not willing to host a beehive fence, a random allocation of treatments was not possible.…”

Dealing with false positive risk as an indicator of misperceived effectiveness of conservation interventions

“…The use of p values is ubiquitous and the replacement of this practice is not realistic, at least in effectiveness studies where null hypothesis significance testing reliant on p values is still the main approach. Even though the use of effect size metrics like relative risk, odds ratio, magnitude of change and Hedge’s d becomes more common to estimate effectiveness [ 10 ], testing can be useful to provide additional support for conclusions. Application of information-theoretic approaches using Akaike Information Criterion (AIC) and similar criteria is the most popular alternative to the use of p values in ecology, and the interest in Bayesian methods is increasing [ 26 , 28 , 38 ].…”

“…The examples of conservation interventions are many, from locally applied electric fences to reduce damage or boost survival of fenced populations [ 6 – 8 ] to globally important protected areas aimed at curbing biodiversity loss [ 9 ]. Reduction-aimed interventions strive to reduce negative outcomes, such as poaching or damage by wildlife, and addition-aimed interventions are used to increase positive outcomes, such as species survival or richness [ 10 ]. Selection of most effective interventions and their wide applications are pivotal to build bridges between science and conservation and to foster good practices [ 11 ].…”

Defining practical and robust study designs for interventions targeted at terrestrial mammalian predators

“…12 interval in relation to no effect (when β = 0, odds ratio = 1), e.g. odds ratio = 0.4 means a decrease of effect by 60% (Khorozyan 2020). I ranked multinomial models according to their Akaike Information Criterion (AIC) and selected the best models as those having ΔAIC < 2 and the highest model weights towards 1 (Symonds & Moussali 2011).…”