Deciphering ecology from statistical artefacts: Competing influence of sample size, prevalence and habitat specialization on species distribution models and how small evaluation datasets can inflate metrics of performance

Abstract

Aim Sample size and species characteristics, including prevalence and habitat specialization, can influence the predictive performance of species distribution models (SDMs). There is little agreement, however, on which metric of model performance to use. Here, we directly compare AUC and partial ROC as metrics of SDM performance through analyses on the effects of species traits and sample size on SDM performance. Location Three counties dominated by agricultural lands and coniferous forest in Oregon's Willamette Valley and Coast Range ecoregions. Methods We systematically reduced a large avian point count dataset to alter sample sizes of 22 species of songbird. We used boosted regression trees to run SDMs for each species, quantified habitat specialization, and used mixed effects models to compare the influence of sample size, prevalence, and habitat specialization on SDM performance, calculated as AUC and partial ROC, across species. We calculated AUC and partial ROC with subset and independent evaluation data separately to more comprehensively investigate differences in metrics. Results We found a positive quadratic effect of sample size and a strongly positive effect of habitat specialization on both metrics of model performance. We found a weak effect of prevalence on partial ROC and no effect in AUC. Contrary to expectations, when evaluated with a subset evaluation data, partial ROC was consistently highest in models with the smallest sample sizes. These small sample size models had correspondingly small sample sizes in subset evaluation datasets. Partial ROC evaluated with independent data and AUC evaluated with subset or independent data showed the expected positive correlation between sample size and model performance. Main Conclusions We found that small evaluation datasets can artificially inflate partial ROC. With literature recommended minimum SDM sample sizes as low as three, attention must be given to the effects of correspondingly low sample sizes in evaluation datasets.

Evaluation metrics and validation of presence-only species distribution models based on distributional maps with varying coverage

“…In our case, the metrics that showed the highest agreement with the ground-truth evaluation were AUC, TSS, MAE, Bias, and ECE. When using datasets with small spatial coverage, care must be taken to provide an adequate number of samples for evaluation, as metrics may be dependent on sample size 81 . In our study, it was shown that AUC and TSS are higher in datasets containing more points.…”

The uniform AUC: Dealing with the representativeness effect in presence–absence models

“…Besides sample size and coverage, other sources of uncertainty and error in the data must be carefully controlled for (see a review of main problems and possible solutions in Rocchini et al, 2011). In general, with a poor‐quality evaluation dataset, the reliability of the models will always be uncertain (Hallman & Robinson, 2020; Jiménez‐Valverde, 2020, 2021).…”

Grand challenges at the frontiers of bird science

“…Current evidence is reasonably clear that careful filtering to select data only most relevant to a particular research question and only from the most experienced observers produces results that align well with results generated from highly trained professionals (Steen et al, 2019). In some cases, more than 90% of community data goes unused after the filtering process (Hallman and Robinson, 2020b). Yet those 90% can be used for thousands of other questions, such as learning how birders improve their skills through time, how the adoption of and contributions to community science differ across geography, and other aspects of the human dimensions of biodiversity interests.…”