Dissecting Machine-Learning Prediction of Molecular Activity: Is an Applicability Domain Needed for Quantitative Structure–Activity Relationship Models Based on Deep Neural Networks?

Abstract

Deep neural networks (DNNs) are the major drivers of recent progress in artificial intelligence. They have emerged as the machine-learning method of choice in solving image and speech recognition problems, and their potential has raised the expectation of similar breakthroughs in other fields of study. In this work, we compared three machine-learning methods DNN, random forest (a popular conventional method), and variable nearest neighbor (arguably the simplest method)in their ability to predict the molecular activities of 21 in vivo and in vitro data sets. Surprisingly, the overall performance of the three methods was similar. For molecules with structurally close near neighbors in the training sets, all methods gave reliable predictions, whereas for molecules increasingly dissimilar to the training molecules, all three methods gave progressively poorer predictions. For molecules sharing little to no structural similarity with the training molecules, all three methods gave a nearly constant valueapproximately the average activity of all training moleculesas their predictions. The results confirm conclusions deduced from analyzing molecular applicability domains for accurate predictions, i.e., the most important determinant of the accuracy of predicting a molecule is its similarity to the training samples. This highlights the fact that even in the age of deep learning, developing a truly high-quality model relies less on the choice of machine-learning approach and more on the availability of experimental efforts to generate sufficient training data of structurally diverse compounds. The results also indicate that the distance to training molecules offers a natural and intuitive basis for defining applicability domains to flag reliable and unreliable quantitative structure−activity relationship predictions.

Learning from Failure: Predicting Electronic Structure Calculation Outcomes with Machine Learning Models

“…Feature space distance metrics 5,61,70,72,[110][111][112] have previously been motivated as a potential uncertainty measure, but because we did not carry out feature selection 60 in this work, the utility of feature space distances for estimating prediction uncertainty is limited (Supporting Information Figure S7). We thus introduce an uncertainty measure that depends directly on the data distribution in the ANN latent space 113 , i.e., the space spanned by the last layer of neurons before the output layer.…”

Drug discovery with explainable artificial intelligence

“…Most of the deep learning models in drug discovery currently do not consider applicability domain restrictions 166,167 , that is, the region of chemical space where statistical learning assumptions are met. These restrictions should, in the authors' opinion, be considered an integral element of XAI, as their assessment and a rigorous evaluation of model accuracy has proven to be more relevant for decision-making than the modelling approach itself 168 . Knowing when to apply which particular model will probably help address the problem of high confidence of deep learning models on wrong predictions 121 and avoid unnecessary extrapolations at the same time.…”

In silico toxicology: From structure–activity relationships towards deep learning and adverse outcome pathways

“…In addition to the question of how an applicability domain can be successfully implemented, it is also interesting to see which models need a defined applicability domain. Liu and coworkers showed that, while it is generally accepted that traditional machine learning models need an applicability domain, deep learning models also cannot generalize much better . However, this is not very surprising given that until now most deep learning models are not built on much larger datasets than traditional models.…”