Estimating the Area Under ROC Curve When the Fitted Binormal Curves Demonstrate Improper Shape

Abstract

Rationale and Objectives The “binormal” model is the most frequently used tool for parametric receiver operating characteristic (ROC) analysis. The binormal ROC curves can have “improper” (non-concave) shapes that are unrealistic in many practical applications, and several tools (eg, PROPROC) have been developed to address this problem. However, due to the general robustness of binormal ROCs, the improperness of the fitted curves might carry little consequence for inferences about global summary indices, such as the area under the ROC curve (AUC). In this work, we investigate the effect of severe improperness of fitted binormal ROC curves on the reliability of AUC estimates when the data arise from an actually proper curve. Materials and Methods We designed theoretically proper ROC scenarios that induce severely improper shape of fitted binormal curves in the presence of well-distributed empirical ROC points. The binormal curves were fitted using maximum likelihood approach. Using simulations, we estimated the frequency of severely improper fitted curves, bias of the estimated AUC, and coverage of 95% confidence intervals (CIs). In Appendix S1, we provide additional information on percentiles of the distribution of AUC estimates and bias when estimating partial AUCs. We also compared the results to a reference standard provided by empirical estimates obtained from continuous data. Results We observed up to 96% of severely improper curves depending on the scenario in question. The bias in the binormal AUC estimates was very small and the coverage of the CIs was close to nominal, whereas the estimates of partial AUC were biased upward in the high specificity range and downward in the low specificity range. Compared to a non-parametric approach, the binormal model led to slightly more variable AUC estimates, but at the same time to CIs with more appropriate coverage. Conclusions The improper shape of the fitted binormal curve, by itself, ie, in the presence of a sufficient number of well-distributed points, does not imply unreliable AUC-based inferences.

The relevance of Short-Term Variation (STV) value measured within 1 hour before delivery in predicting adverse neonatal outcome

“…Area under curve (AUC) was calculated for the study groups for Apgar score in 1 st and 5 th minutes. In Group 1 the AUC measurements were 0.45 (95% CI: 0.32-0.58) for first minute and 0.55 (95% CI: 0.35-0.74) for fifth minute and in Group 2: 0.58 (95% CI: 0.45-0.71) for the first minute and 0.57 (95% CI: 0.42-0.72) for the fifth minute [8].…”

A 17-Gene Signature Predicted Prognosis in Renal Cell Carcinoma

“…One of the most commonly used ROC summary indices was the area under the ROC curve (AUC). The AUC values were calculated from the ROC curve [28,29]. p < 0:05 was considered to be statistically significant.…”

Affinity-based measures of biomarker performance evaluation

“…We consider such settings for full generality but note that while they may be meaningful from a conceptual perspective, they may not necessarily be sensible for many practical settings—as they would correspond to locally worse than chance performance. 21,22,32–34…”