General Approach to Estimate Error Bars for Quantitative Structure–Activity Relationship Predictions of Molecular Activity

Abstract

Key requirements for quantitative structure-activity relationship (QSAR) models to gain acceptance by regulatory authorities include a defined domain of applicability (DA) and appropriate measures of goodness-of-fit, robustness, and predictivity. Hence, many DA metrics have been developed over the past two decades. The most intuitive are perhaps distance-to-model metrics, which are most commonly defined in terms of the mean distance between a molecule and its k nearest training samples. Detailed evaluations have shown that the variance of predictions by an ensemble of QSAR models may serve as a DA metric and can outperform distance-to-model metrics. Intriguingly, the performance of ensemble variance metric has led researchers to conclude that the error of predicting a new molecule does not depend on the input descriptors or machine-learning methods but on its distance to the training molecules. This implies that the distance to training samples may serve as the basis for developing a high-performance DA metric. In this article, we introduce a new Tanimoto distance-based DA metric called the sum of distance-weighted contributions (SDC), which takes into account contributions from all molecules in a training set. Using four acute chemical toxicity data sets of varying sizes and four other molecular property data sets, we demonstrate that SDC correlates well with the prediction error for all data sets regardless of the machine-learning methods and molecular descriptors used to build the QSAR models. Using the acute toxicity data sets, we compared the distribution of prediction errors with respect to SDC, the mean distance to k-nearest training samples, and the variance of random forest predictions. The results showed that the correlation with the prediction error was highest for SDC. We also demonstrate that SDC allows for the development of robust root mean squared error (RMSE) models and makes it possible to not only give a QSAR prediction but also provide an individual RMSE estimate for each molecule. Because SDC does not depend on a specific machine-learning method, it represents a canonical measure that can be widely used to estimate individual molecule prediction errors for any machine-learning method.

A quantitative uncertainty metric controls error in neural network-driven chemical discovery

“…To demonstrate the advantages of the latent space distance metric in a quantitative fashion, we compare to three established uncertainty metrics. This assessment is particularly motivated by the nature of chemical discovery applications,8 where data set sizes are often smaller and have more broadly varying chemistry than typical applications in neural network potentials19,40 or in quantitative structure–property relationships in cheminformatics 41,52. To mimic chemical discovery efforts, we train neural networks to predict transition metal complex spin state energetics7 and test them on diverse transition metal complexes from experimental databases.…”

“…A final class of widely applied uncertainty metrics employs distances in feature space of the test molecule to available training data to provide an estimate of molecular similarity and thus model applicability. The advantages of feature space distances are that they are easily interpreted, may be rapidly computed, and are readily applied regardless of the regression model7,8,41,52 (Fig. 1).…”

Learning from Failure: Predicting Electronic Structure Calculation Outcomes with Machine Learning Models

“…Feature space distance metrics 5,61,70,72,[110][111][112] have previously been motivated as a potential uncertainty measure, but because we did not carry out feature selection 60 in this work, the utility of feature space distances for estimating prediction uncertainty is limited (Supporting Information Figure S7). We thus introduce an uncertainty measure that depends directly on the data distribution in the ANN latent space 113 , i.e., the space spanned by the last layer of neurons before the output layer.…”

Uncertain Times Call for Quantitative Uncertainty Metrics: Controlling Error in Neural Network Predictions for Chemical Discovery

“…To demonstrate the advantages of the latent space distance metric in a quantitative fashion, we compare to three established uncertainty metrics. This assessment is particularly motivated by the nature of chemical discovery applications 8,25 , where data set sizes are often smaller and have more broadly varying chemistry than typical applications in neural network potentials 19,41 and in quantitative structure-property relationships in cheminformatics 42,52 . To mimic chemical discovery efforts, we train neural networks to predict transition metal complex spin state energetics 7 and test them on diverse transition metal complexes from experimental databases.…”

“…A final class of widely applied uncertainty metrics employs distances in feature space of the test molecule to available training data to provide an estimate of molecular similarity and thus model applicability. The advantages of feature space distances are that they are easily interpreted, may be rapidly computed, and are readily applied regardless of the regression model 7,25,42,52 (Figure 1). We used [7][8]25 high feature space distances to successfully reduce model prediction errors on retained points while still discovering new transition metal complexes.…”