Two New Parameters Based on Distances in a Receiver Operating Characteristic Chart for the Selection of Classification Models

Abstract

There are several indices that provide an indication of different types on the performance of QSAR classification models, being the area under a Receiver Operating Characteristic (ROC) curve still the most powerful test to overall assess such performance. All ROC related parameters can be calculated for both the training and test sets, but, nevertheless, neither of them constitutes an absolute indicator of the classification performance by themselves. Moreover, one of the biggest drawbacks is the computing time needed to obtain the area under the ROC curve, which naturally slows down any calculation algorithm. The present study proposes two new parameters based on distances in a ROC curve for the selection of classification models with an appropriate balance in both training and test sets, namely the following: the ROC graph Euclidean distance (ROCED) and the ROC graph Euclidean distance corrected with Fitness Function (FIT(λ)) (ROCFIT). The behavior of these indices was observed through the study on the mutagenicity for four genotoxicity end points of a number of nonaromatic halogenated derivatives. It was found that the ROCED parameter gets a better balance between sensitivity and specificity for both the training and prediction sets than other indices such as the Matthews correlation coefficient, the Wilk's lambda, or parameters like the area under the ROC curve. However, when the ROCED parameter was used, the follow-on linear discriminant models showed the lower statistical significance. But the other parameter, ROCFIT, maintains the ROCED capabilities while improving the significance of the models due to the inclusion of FIT(λ).

Exploring simple, transparent, interpretable and predictive QSAR models for classification and quantitative prediction of rat toxicity of ionic liquids using OECD recommended guidelines

“…In order to evaluate the robustness and predictivity, the developed models were subjected to validation employing multiple strategies and the results are expressed in the form of various metrics Fawcett, 2006;Gálvez et al, 1996;Golbraikh and Tropsha, 2002;Hawkins, 2004;Mahalanobis, 1936;Matthews, 1975;Mitra et al, 2010;Ojha et al, 2011;Pérez-Garrido et al, 2011;Prado-Prado et al, 2009;Roy et al, 2013;Schüürmann et al, 2008;Shi et al, 2001;Snedecor and Cochran, 1967;Wilks, 1932) (see Supplementary Materials). For both the classification and regression-based analyses, the applicability domain of the models was determined using two different distance based approaches: leverage analysis (Atkinson, 1985;Gramatica, 2007) and the mean Euclidean distance (Golmohammadi et al, 2012).…”

Prediction of Milk/Plasma Concentration Ratios of Drugs and Environmental Pollutants Using In Silico Tools: Classification and Regression Based QSARs and Pharmacophore Mapping

“…The ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5. In this study we have calculated two new additional parameters [42] namely the ROC graph Euclidean distance (ROCED) and the ROC graph Euclidean distance corrected with Fitness Function (FIT(l)) (ROCFIT) to have better explainable results. In the present study, we have used another validation diagram named PDD [43] to verify the degree and extent of discrimination achieved in the training as well as test set observations.…”

Prediction of Th Cell Epitopes Based on ISC-SVR Method

“…详细实现流程如下: AUC [24,25] (the area under the receiver operator char- [19] 其他预测模型 进行比较结果, 见表 3. 从表 3 可以看出, 本文方法所建 立的预测模型明显优于文献 [19] 中其他方法.…”