A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval

Abstract

Motivation:The performance of classifiers is often assessed using Receiver Operating Characteristic ROC [or (AC) accumulation curve or enrichment curve] curves and the corresponding areas under the curves (AUCs). However, in many fundamental problems ranging from information retrieval to drug discovery, only the very top of the ranked list of predictions is of any interest and ROCs and AUCs are not very useful. New metrics, visualizations and optimization tools are needed to address this 'early retrieval' problem. Results: To address the early retrieval problem, we develop the general concentrated ROC (CROC) framework. In this framework, any relevant portion of the ROC (or AC) curve is magnified smoothly by an appropriate continuous transformation of the coordinates with a corresponding magnification factor. Appropriate families of magnification functions confined to the unit square are derived and their properties are analyzed together with the resulting CROC curves. The area under the CROC curve (AUC[CROC]) can be used to assess early retrieval. The general framework is demonstrated on a drug discovery problem and used to discriminate more accurately the early retrieval performance of five different predictors. From this framework, we propose a novel metric and visualization-the CROC(exp), an exponential transform of the ROC curve-as an alternative to other methods. The CROC(exp) provides a principled, flexible and effective way for measuring and visualizing early retrieval performance with excellent statistical power. Corresponding methods for optimizing early retrieval are also described in the Appendix. Availability: Datasets are publicly available. Python code and command-line utilities implementing CROC curves and metrics are available at

Deep Learning to Predict the Formation of Quinone Species in Drug Metabolism

“…Paired t -tests 54 indicated that the neural network and the logistic regressor were statistically equivalent for both average pair AUC ( p -value 0.339) and top-two ( p -value 0.286). To break the tie, we constructed reliability plots, which quantify how well predictions correlate to probabilities (Figure 5).…”

“…However, both the pair-level neural network and the pair-level logistic regressor significantly improved performance compared to only training on the atom-level, especially by the top-two metric. For example, comparing the top-two performances of the pair-level logistic regressor and the atom-level neural network by a paired t -test 54 yielded a p -value of 0.037. Consequently, we retained the pair-level training and model.…”

“…Using 10-fold cross-validation, neither the neural network nor the logistic regressor outperformed the maximum pair-level score, with molecule AUCs of 88.2% ( p -value 0.16) and 88.4% ( p -value 0.31), respectively, with significance evaluated by a false positive rate paired t -test. 54 …”

Structure based inhibitor design targeting glycogen phosphorylase b. Virtual screening, synthesis, biochemical and biological assessment of novel N-acyl-β-d-glucopyranosylamines

“…For the training set, performance with respect to recovery of the actives (top 10 inhibitors) was analyzed in a statistical manner [24][25][26][27] .…”

Design and Analysis of Classifier Learning Experiments in Bioinformatics: Survey and Case Studies

“…Through a simulation study and analysis of real microarray data, they show that the difference is considerable. Swamidass et al [25] propose the concentrated ROC framework in which any relevant portion of the ROC curve is magnified smoothly by an appropriate continuous transformation. The area under the ROC curve assesses retrieval performance of the relevant portion.…”