Efficiency of different measures for defining the applicability domain of classification models

Abstract

The goal of defining an applicability domain for a predictive classification model is to identify the region in chemical space where the model’s predictions are reliable. The boundary of the applicability domain is defined with the help of a measure that shall reflect the reliability of an individual prediction. Here, the available measures are differentiated into those that flag unusual objects and which are independent of the original classifier and those that use information of the trained classifier. The former set of techniques is referred to as novelty detection while the latter is designated as confidence estimation. A review of the available confidence estimators shows that most of these measures estimate the probability of class membership of the predicted objects which is inversely related to the error probability. Thus, class probability estimates are natural candidates for defining the applicability domain but were not comprehensively included in previous benchmark studies. The focus of the present study is to find the best measure for defining the applicability domain for a given binary classification technique and to determine the performance of novelty detection versus confidence estimation. Six different binary classification techniques in combination with ten data sets were studied to benchmark the various measures. The area under the receiver operating characteristic curve (AUC ROC) was employed as main benchmark criterion. It is shown that class probability estimates constantly perform best to differentiate between reliable and unreliable predictions. Previously proposed alternatives to class probability estimates do not perform better than the latter and are inferior in most cases. Interestingly, the impact of defining an applicability domain depends on the observed area under the receiver operator characteristic curve. That means that it depends on the level of difficulty of the classification problem (expressed as AUC ROC) and will be largest for intermediately difficult problems (range AUC ROC 0.7–0.9). In the ranking of classifiers, classification random forests performed best on average. Hence, classification random forests in combination with the respective class probability estimate are a good starting point for predictive binary chemoinformatic classifiers with applicability domain.Graphical abstract. Electronic supplementary materialThe online version of this article (doi:10.1186/s13321-017-0230-2) contains supplementary material, which is available to authorized users.

Chemogenomic Active Learning's Domain of Applicability on Small, Sparse qHTS Matrices: A Study Using Cytochrome P450 and Nuclear Hormone Receptor Families

“…Confidence estimation is defined as the probability of a model to misclassify a new entity. For random forests, confidence can be estimated based on the variance of prediction among the single decision trees in a forest . The metric for the number of single models (trees) that agree with the combined model (random forest) has been termed concordance .…”

Prediction of hERG K+ channel blockage using deep neural networks

“…The final result is the average of individual trees. Random forest can handle both regression problems and classification problems (Chen et al, ; Da et al, ; Klingspohn et al, ). There are a handful of adjustable parameters (e.g., number of trees, maximum depth of the tree, number of features), and these parameters have significant influence on the performance of the algorithm.…”

“…Random forest method is one of the widely used machine learning techniques employed in pharmaceutical research (Chen, Sheridan, Hornak, & Voigt, 2012;Da, Desaphy, Bret, & Rognan, 2015;Klingspohn, Mathea, Ter Laak, Heinrich, & Baumann, 2017;Marchese Robinson, Palczewska, Palczewski, & Kidley, 2017). It is an ensemble prediction model, which fits many decision trees trained on subsampled subsets of the original data set (Svetnik et al, 2003).…”

A combined drug discovery strategy based on machine learning and molecular docking

“…Researchers have developed various algorithms, such as decision tree (DT), k-Nearest neighbor (kNN), support vector machine (SVM), random forest (RF), neural network etc. to achieve desirable outcomes in decision classification problems (Klingspohn, Mathea, Ter Laak, Heinrich, & Baumann, 2017;Podgorelec, Kokol, Stiglic, & Rozman, 2002;Sprague et al, 2014;Tian, Wang, Li, Xu, & Hou, 2012). For example, the classification and regression models are widely used in the discrimination of actives and inactives for a certain target (Hammann, Gutmann, Baumann, Helma, & Drewe, 2009;Helma, Cramer, Kramer, & De Raedt, 2004).…”

“…In terms of drug discovery, RF has become a common standard for quantitative predictive modeling (Chen, Sheridan, Hornak, & Voigt, 2012;Da, Desaphy, Bret, & Rognan, 2015). Several studies (Chen et al, 2012;Da et al, 2015;Klingspohn et al, 2017) have shown that RF models perform well in solving problems related to predictive chemical information. It possesses great advantages in solving classification problems.…”