Goal-oriented evaluation of species distribution models’ accuracy and precision: True Skill Statistic profile and uncertainty maps

Abstract

The use of species distribution models’ (SDM) is limited by its performance in terms of accuracy, precision, or the spatial distribution of model errors. Despite the wide acceptance of some standard statistics used to evaluate SDM, there is currently a strong on-going debate as to their use. The “area under the curve” (AUC) is a popular measure used to evaluate SDMs; however, it does not provide complete information about model accuracy. The maximum True Skill Statistic (TSS) is another statistic that is gaining acceptance. However, evaluations of a model’s accuracy solely based on this statistic may also be misleading. We investigate the use of alternative methods to evaluate the performance of SDMs, to objectively compare among different modelling approaches. We evaluate the performance of SDMs fitted to simulated and real data by contrasting model predictions to additional validation datasets. We propose visualising TSS scores over the whole detection threshold range (TSS profile). We show how models with similarly good performance according to AUC, present very different results and may serve to different purposes. Also, a high maximum TSS may not guarantee accurate predictions and should be accompanied by the threshold where the maximum is reached (t*). We observe that the higher t* the better predicted observations correlate with confirmed observations. Also, SDM predictions should be accompanied with the corresponding uncertainty map to avoid misleading conclusions. Too high or too widely spread uncertainty on such maps would question the overall accuracy of the model. Whether the model is intended to detect all potential observation sites (sensitive model) or to accurately predict where confirmed observations could be found (specific model) sets a different performance targets to be achieved by the model. The approach proposed helps to discern which SDM may best suit the intended goals. Furthermore, the TSS profile helps i) to evaluate the overall performance of SDMs and compare among them, ii) to identify the main source of error, and iii) to select a detection threshold depending on the maps intended use.

Effects of Climate Change on the Climatic Niches of Warm-Adapted Evergreen Plants: Expansion or Contraction?

“…Then, the probability of occurrence at which the TSS value was maximized was selected as a threshold for each species (equivalent to maximizing the sum of sensitivity and specificity) [68], one of the best methods to minimize commission and omission errors [69,70]. A total of 20 suitability maps were projected under 20 climate change scenarios for each species for the given period, 2050 and 2070.…”

South Africa’s newly approved marine protected areas have increased the protected modelled habitat of nine odontocete species

“…All 9 odontocete species had TSS and AUC scores > 0.90, indicating an excellent discriminatory ability (Pennino et al 2017, Pereira et al 2018. TSS scores, although lower in our results, have better model accuracy evaluation for species where there are no true absences (Allouche et al 2006, Ruete & Leynaud 2015, Shabani et al 2016. This is because the TSS method of evaluation is insensitive and independent of prevalence, meaning it is able to calculate the percentage of accurately predicted presences and absences (Allouche et al 2006, Ruete & Leynaud 2015, Shabani et al 2016).…”

“…This is because the TSS method of evaluation is insensitive and independent of prevalence, meaning it is able to calculate the percentage of accurately predicted presences and absences (Allouche et al 2006, Ruete & Leynaud 2015, Shabani et al 2016). On the other hand, the AUC method only evaluates sensitivity (predicted presences) and not the specificity (predicted absences) of the model (Allouche et al 2006, Ruete & Leynaud 2015, Shabani et al 2016.…”

Classification of unlabeled observations in Species Distribution Modelling using Point Process Models

“…Although PA data is generally of higher quality, it is also less common than PO data because it requires more rigorous planning to visit a set of pre-determined sites. On the other hand, PO data sets are very common, arising from surveys or opportunistic sightings, but they usually have lower quality (van Strien et al , 2013; Ruete & Leynaud, 2015). Point process models (PPMs) are a common tool for fitting SDMs to analyze PO data (Warton & Shepherd, 2010; Mi et al , 2014; Renner et al , 2015) and have been used to fit models for real datasets and simulated data (Baddeley et al , 2006; Illian et al , 2012; Renner & Warton, 2013; Baddeley et al , 2015).…”