Applicability Domains for Classification Problems: Benchmarking of Distance to Models for Ames Mutagenicity Set

Abstract

The estimation of accuracy and applicability of QSAR and QSPR models for biological and physicochemical properties represents a critical problem. The developed parameter of "distance to model" (DM) is defined as a metric of similarity between the training and test set compounds that have been subjected to QSAR/QSPR modeling. In our previous work, we demonstrated the utility and optimal performance of DM metrics that have been based on the standard deviation within an ensemble of QSAR models. The current study applies such analysis to 30 QSAR models for the Ames mutagenicity data set that were previously reported within the 2009 QSAR challenge. We demonstrate that the DMs based on an ensemble (consensus) model provide systematically better performance than other DMs. The presented approach identifies 30-60% of compounds having an accuracy of prediction similar to the interlaboratory accuracy of the Ames test, which is estimated to be 90%. Thus, the in silico predictions can be used to halve the cost of experimental measurements by providing a similar prediction accuracy. The developed model has been made publicly available at http://ochem.eu/models/1 .

QSAR models and scaffold-based analysis of non-nucleoside HIV RT inhibitors

“…In the current study, the CONSENSUS-STD (standard deviation of predictions of the ensemble of models in the consensus model) was used as a measure of DM. This DM provided the best separation of molecules with low and high accuracy of predictions in several benchmarking studies [59,60]. A threshold value of 95% of compounds from the training set was used to determine the qualitative ADs of models.…”

Mixed learning algorithms and features ensemble in hepatotoxicity prediction

“…One may also generate many models from the same training set but using different subset of features (F var ). Most of these approaches have been explored, but frequently used were ensemble of fixed learning algorithm with varied features and (or) varied training set (T fix Al fix F var or T var Al fix F var ) [35,36,[40][41][42][43][44][45][46][47][48]. For example, a technique that uses mixed training set and features (T var Al fix F var ) is the Random Forest [49] technique which is an ensemble of many decision trees built from a variation of sample and features.…”

The Challenges Involved in Modeling Toxicity Data In Silico: A Review

“…Whilst this may be interpreted as a range of chemical structures for which the expected model performance is well characterized [180], the "applicability domain" is commonly interpreted as a region of chemical structure space in which the model is known to exhibit desirable predictivity [155, 156, 158 205 208]. A distinction may be made between those approaches which simply try to categorize compounds as inside the applicability domain (AD) or outside the AD, and those which seek to directly assess the expected performance of the model for a particular compound [205]. In the context of predictive toxicology, where the "mechanism of toxic action" is understood, the former approach may be informed by mechanistic reasoning [207].…”