The uniform AUC: Dealing with the representativeness effect in presence–absence models

Abstract

Species distribution models use occurrence data of the focus species together with environmental variables to estimate a habitat suitability score. Typically, the discrimination capacity of the models, that is, how good the instances of presence and absence are correctly classified, is assessed with statistics such as the area under the receiver operating characteristic curve (AUC). However, the value of any discrimination statistic depends on the distribution of the suitability scores in the evaluation dataset, the so‐called representativeness effect. For example, the same well‐calibrated model evaluated with two datasets, one set with mostly intermediate predicted values (around 0.5) and another set in which most cases have extreme predicted values (close to 0 and close to 1), will yield low (towards 0.5) and high (towards 1) AUC values, respectively. Thus, discrimination values are entirely context dependent and cannot be directly compare between datasets because they are not intrinsic measures of a model’s performance. In this contribution, I propose a methodology based on stratified bootstrapping with an inverse probability weighting that makes the distribution of the suitability values uniform and harmonizes the AUC so that the so‐called uAUC can now be compared between different datasets. I run simulations considering a range of sample sizes and different initial distributions of the suitability values to validate the method. I apply the method to an empirical exercise, and implement it in the R package vandalico. Although I mainly focused on the AUC, the harmonization procedure can be applied to any other discrimination measure that is prevalence independent in a strict mathematical sense, such as the sensitivity star (Se*). To provide accurate estimations, it is important to have a high‐quality evaluation dataset, that is, a dataset that covers the whole spectrum of suitability values and a minimum sample size of at least 150 cases. Further research is needed to develop a method to estimate reliable confidence intervals. I expect the harmonization methodology proposed here to be of interest in other research areas (besides ecology) that deal with discrimination/classification problems.