Plotting receiver operating characteristic and precision-recall curves from presence and background data

Abstract

1. The receiver operating characteristic (ROC) and precision-recall (PR) plots have been widely used to evaluate the performances of species distribution models. Plotting ROC/PR curves requires a traditional test set with both presence and absence data (namely PA approach), but species absence data are usually not available in reality. Plotting ROC/PR curves from presence-only data while treating background data as pseudo absence data (namely PO approach) may provide misleading results. 2. In this study we propose a new approach to calibrate the ROC/PR curves from presence and background data with user-provided information on a constant c, namely PB approach. An estimate of c can also be derived from the PB-based ROC/PR plots given that a model with good ability of discrimination is available. We used three virtual species and a real aerial photography to test the effectiveness of the proposed PB-based ROC/PR plots. Different models (or classifiers) were trained from presence and background data with various samples sizes. The ROC/PR curves plotted by PA approach were used to benchmark the curves plotted by PO and PB approaches. 3. Experimental results show that the curves and areas under curves by PB approach are more similar to that by PA approach as compared with PO approach. The PB-based ROC/PR plots also provide highly accurate estimations of c in our experiment. 4. We conclude that the proposed PB-based ROC/PR plots can provide valuable complements to existing model assessment methods, and they also provide an additional way to estimate the constant c (or species prevalence) from presence and background data.

Revisiting sample size planning for receiver operating characteristic studies: A confidence interval approach with precision and assurance

“…Under a commonly used assumption that data are normally distributed, a variance function has previously been derived, 3,8,9 with f false( θ false) given bywhere ϕ − 1 false( ⋅ false) is the inverse of the cumulative distribution function of the standard normal distribution and r is the sample size ratio of the control group to the disease group. This function, together with a less frequently used version that additionally involves a standard deviation ratio parameter, tends to provide conservative sample size estimates 3,8,21 ; they were originally derived to improve on the variance based on an exponential model, 1 which can underestimate the sample size when the disease and non-disease groups have unequal variances of observations. 8 While a conservative estimate is useful to reduce sample size underestimation, being conservative comes from an overestimation of the true variance.…”

“…1,5,8–18 Here we focus on a different criterion, whether the estimates are of acceptable precision, as quantified by the lower confidence limit. 19–21 Note that hypothesis testing focuses on a single value (the null hypothesis) while CI estimation focuses on a range of plausible values and thus is more informative than the former approach. 22 Based on benchmarks 23,24 such as an AUC between 0.9 and 1.0 reflects “excellent” accuracy, 0.8 and 0.9 “good” accuracy, 0.7 and 0.8 “fair” accuracy, 0.6 and 0.7 “poor” accuracy, and below 0.6 “failed”, one might ask “how many participants should be recruited to assure with 90% chance that the lower confidence limit is above 0.8 (so that I feel comfortable to call the test good)?”…”

“…Application of the delta method 19,21 with a logit transformation for AUC yields a general sample size formula as where the logit transformation gives logit false( θ false) = log falsefalse{ θ / ( 1 − θ ) falsefalse}, z β is the upper β quantile of the standard normal distribution, 1 − α is the confidence level, and f false( θ false) is the kernel of the variance for the estimated AUC (factoring out the sample size), that is, f false( θ false) / n = var false( θ false^ false) .…”