Comparison of Scaling Methods to Obtain Calibrated Probabilities of Activity for Protein–Ligand Predictions

Abstract

In the context of bioactivity prediction, the question of how to calibrate a score produced by a machine learning method into a probability of binding to a protein target is not yet satisfactorily addressed. In this study, we compared the performance of three such methods, namely, Platt scaling (PS), isotonic regression (IR), and Venn–ABERS predictors (VA), in calibrating prediction scores obtained from ligand–target prediction comprising the Naïve Bayes, support vector machines, and random forest (RF) algorithms. Calibration quality was assessed on bioactivity data available at AstraZeneca for 40 million data points (compound–target pairs) across 2112 targets and performance was assessed using stratified shuffle split (SSS) and leave 20% of scaffolds out (L20SO) validation. VA achieved the best calibration performances across all machine learning algorithms and cross validation methods tested and also the lowest (best) Brier score loss (mean squared difference between the outputted probability estimates assigned to a compound and the actual outcome). In comparison, the PS and IR methods can actually degrade the assigned probability estimates, particularly for the RF for SSS and during L20SO. Sphere exclusion, a method to sample additional (putative) inactive compounds, was shown to inflate the overall Brier score loss performance, through the artificial requirement for inactive molecules to be dissimilar to active compounds, but was shown to result in overconfident estimators. VA was able to successfully calibrate the probability estimates for even small calibration sets. The multiprobability values (lower and upper probability boundary intervals) were shown to produce large discordance for test set molecules that are neither very similar nor very dissimilar to the active training set, which were hence difficult to predict, suggesting that multiprobability discordance can be used as an estimate for target prediction uncertainty. Overall, we were able to show in this work that VA scaling of target prediction models is able to improve probability estimates in all testing instances and is currently being applied for in-house approaches.

Computational analyses of mechanism of action (MoA): data, methods and integration

“…224 Despite the advantages of using CNNs, it is important to keep in mind that their performance could be limited by the data availability as often imaging datasets are small and heavily conditional. 225 In fact, other important model characteristics such as the applicability domain (where the model works with high reliability and where it doesn't, for example in terms of areas of new chemical space, e.g., Reliability Density Neighbourhoods 226 ) and prediction uncertainty (Venn-Abers, conformal prediction) 227,228 should also be considered, as well as performance-based measures such as accuracy, AUROC and AUPRC, but are often neglected in bioactivity model evaluations despite providing a measure of how confident one can be in new predictions (which is the ultimate goal of target prediction, and any supervised ML model).…”

In silico prediction and biological assessment of novel angiogenesis modulators from traditional Chinese medicine

“…Ligand-based target prediction methods rely on the principle of chemical similarity, which assumes that compounds with similar chemical structure should exhibit similar biological effects (Mervin et al, 2018a;Mervin et al, 2018b;Mervin et al, 2020). While this principle generally holds across large datasets, it is not always valid, e.g., due to "Activity Cliffs," where the activity of a compound changes abruptly, despite only minor changes in the chemical structure (Young et al, 2008;Stumpfe et al, 2019).…”

Multispecies Machine Learning Predictions of In Vitro Intrinsic Clearance with Uncertainty Quantification Analyses

“…Such uncertainty needs to be incorporated into the decision-making process, for instance, using a binning or categorical system to compare CL int values for molecule prioritization. ,, However, experimental variability estimates (aleatoric or statistical uncertainty) , are typically not accessible in the public domain due to the lack of repeated measurements . Aleatoric or statistical uncertainty is the noise inherent to the experiments, which comes from the data generation, has a stochastic nature, and is irreducible. − Using internal data, experimental variability can be estimated using blind replicates measured over the years. Statistical uncertainty cannot be changed nor reduced and allows estimating the upper performance limits of in silico models. , Uncertainty can also be estimated for model predictions (epistemic uncertainty) to detect compounds outside the applicability domain.…”