Goal-oriented evaluation of species distribution models’ accuracy and precision: True Skill Statistic profile and uncertainty maps

Abstract

The use of species distribution models’ (SDM) is limited by its performance in terms of accuracy, precision, or the spatial distribution of model errors. Despite the wide acceptance of some standard statistics used to evaluate SDM, there is currently a strong on-going debate as to their use. The “area under the curve” (AUC) is a popular measure used to evaluate SDMs; however, it does not provide complete information about model accuracy. The maximum True Skill Statistic (TSS) is another statistic that is gaining acceptance. However, evaluations of a model’s accuracy solely based on this statistic may also be misleading. We investigate the use of alternative methods to evaluate the performance of SDMs, to objectively compare among different modelling approaches. We evaluate the performance of SDMs fitted to simulated and real data by contrasting model predictions to additional validation datasets. We propose visualising TSS scores over the whole detection threshold range (TSS profile). We show how models with similarly good performance according to AUC, present very different results and may serve to different purposes. Also, a high maximum TSS may not guarantee accurate predictions and should be accompanied by the threshold where the maximum is reached (t*). We observe that the higher t* the better predicted observations correlate with confirmed observations. Also, SDM predictions should be accompanied with the corresponding uncertainty map to avoid misleading conclusions. Too high or too widely spread uncertainty on such maps would question the overall accuracy of the model. Whether the model is intended to detect all potential observation sites (sensitive model) or to accurately predict where confirmed observations could be found (specific model) sets a different performance targets to be achieved by the model. The approach proposed helps to discern which SDM may best suit the intended goals. Furthermore, the TSS profile helps i) to evaluate the overall performance of SDMs and compare among them, ii) to identify the main source of error, and iii) to select a detection threshold depending on the maps intended use.

Predicting Spatial Patterns of Sindbis Virus (SINV) Infection Risk in Finland Using Vector, Host and Environmental Data

“…We repeated the calibration and evaluation sets 10 times (80 model evaluation runs in total) for vector modeling, and 50 times (400 model evaluation runs in total) for Pogosta disease modeling [79]. The area under the receiver operating characteristic (AUC) value was used to assess the model performance in the analyses; scores range from 0 to 1, with 0.5 being the threshold for predictions better than random [80,81]. Sensitivity (the proportion of observed presences) and specificity (the proportion of observed absences) were calculated to quantify the omission errors [80].…”

Continent-Wide Tree Species Distribution Models May Mislead Regional Management Decisions: A Case Study in the Transboundary Biosphere Reserve Mura-Drava-Danube

“…TSS values range from −1 to +1 and are derived as: sensitivity + specificity − 1. The higher the output value, the better the SDM performance [85]. Cohen's kappa coefficient is considered to correct for the model fit expected by chance [86] and is therefore frequently applied to support model accuracy evaluation [50,53].…”