Causal inference and large-scale expert validation shed light on the drivers of SDM accuracy and variance

Abstract

1.The literature is awash with studies purporting to show how various species and data characteristics affect the performances of Species Distribution Models (SDMs). Many of these studies follow a similar template: they fit SDMs for several species, or the same species using different datasets; assess the accuracy of those SDMs using skill statistics; and then identify correlates thereof. Interpreting the findings of these studies is challenging because skill statistics can reflect species and data characteristics rather than model accuracy, and correlates of model performance are not necessarily causes. 2.Here, we took a different approach to identifying causes of variation in SDM performance. We fitted models for 535 species across 5 invertebrate groups and 1 plant group in the United Kingdom (UK), using a fairly typical SDM workflow. We measured two components of SDM performance: the variance among replicate fits and accuracy. Rather than using skill statistics, accuracy was assessed by taxon experts. We constructed Directed Acyclic Graphs (DAGs) depicting plausible effects of explanatory variables (e.g. species’ prevalence, sample size) on SDM performance, then quantified those effects using multilevel piecewise path models. 3.We found that the degree to which the available data covered species’ environmental niches was the only explanatory variable to affect SDM accuracy. We suggest that previously reported associations between sample size and SDM accuracy reflect improved coverage of species environmental niches at higher sample sizes; that is to say, niche completeness confounds the effect of sample size on SDM accuracy. We also report that the completeness of species’ environmental niches, sample size, species’ prevalence and the degree to which the available data cover species’ geographic ranges affect SDM variance. 4.Our results demonstrate the challenges associated with the high-throughput approach to modelling species’ distributions. There is no guarantee that accurate and precise SDMs can be constructed for large numbers of species unless their ranges and niches have been sampled comprehensively. Decisions about whether modelling is worthwhile should not be based on simple criteria like sample size.

Assessing the impact of binary land cover variables on species distribution models: A North American study on water birds

“…The largest changes in model performance were observed when using smaller sample sizes (Figure 3B), which is in line with previous studies that have explored the impact of sample size on SDMs (Stockwell and Peterson 2002, Wisz et al 2008, Liu et al 2019, Jiménez-Valverde 2020. Additionally, our study's high number of occurrences helps ensure that the data represent most of the species realized niche, which can positively impact model performance regardless of the sample size (Boyd et al 2022). In cases where a smaller sample size is used (tens or hundreds of records), a difference in model performance between models using proportional and binary variables would likely be more pronounced.…”

An operational workflow for producing periodic estimates of species occupancy at national scales

“…For this reason, it would also be useful to consider independent model evaluation using either independent data or elicitation of expert opinion. We are trialling approaches in which data providers are asked to provide opinions about whether model outputs are plausible (Boyd et al ., 2023; Pescott, 2022).…”

Optimising Species Distribution Models: Sample size, positional error, and sampling bias matter

“…For example, if a species has a restricted ecological niche (or range), the niche may likely be well represented by a low number of occurrences. On the other hand, large sample size does not necessarily mean a complete coverage of the entire ecological niche for widespread species (Bazzichetto et al 2023;Boyd et al 2023). This is further related to model complexity.…”

“…As the interest in using SDMs continues to grow, tackling data limitations becomes increasingly critical (Araújo et al 2019;Wüest et al 2020;Jansen et al 2022;Marcer et al 2022). In this context, data characteristics and limitations are expected to be regularly considered and properly reported during the conceptualization and validation of SDMs (Feng et al 2019;Zurell et al 2020;Sillero and Barbosa 2021;Tessarolo et al 2021;Jansen et al 2022;Boyd et al 2023). However, without proper knowledge of the individual or combined effects of sample size, positional error, sampling bias, and species' ecology, our ability to anticipate the effect of these issues on the quality of modelled speciesenvironment associations remains largely uncertain, limiting the value of model outputs (see Figure 1 for a diagram introducing data characteristics and their relationships considered in this perspective).…”