ADMET Predictability at Boehringer Ingelheim: State‐of‐the‐Art, and Do Bigger Datasets or Algorithms Make a Difference?

Aleksić, Stevan; Seeliger, Daniel; Brown, J.

doi:10.1002/minf.202100113

Cited by 28 publications

(45 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, a recent review from Boehringer Ingelheim reported SVM and RF as the top performers on internal ADMET (absorption, distribution, metabolism, excretion, and toxicity) prediction tasks, and furthermore presented mixed results regarding the benefit of increasing training set sizes, with 8 out of 23 datasets recording negative effects on the predictive performance when using more data 13 . The debate of whether more data is helpful for virtual screening, usually framed in the context of virtual docking, has still not been settled 14 .…”

Section: Introductionmentioning

confidence: 99%

Multi-fidelity machine learning models for improved high-throughput screening predictions

Buterez

Janet

Kiddle

et al. 2022

Preprint

View full text Add to dashboard Cite

High throughput screening (HTS) is one of the leading techniques for hit identification in drug discovery and comprises of multiple phases, one primary and one or more confirmatory screens which result in multi-fidelity data. Noisy primary screening data are available on a large number of compounds and higher quality confirmatory data on a low-to-moderate number of compounds. Existing computational pipelines do not integrate primary screening data of individual HTS campaigns, resulting in millions of potentially useful data points being unused for bioactivity prediction. Furthermore, there is a lack of publicly available multi-fidelity bioactivity benchmarks to support modelling real-world HTS data. To address these challenges, we assembled public (PubChem) and private (AstraZeneca) collections of multi-fidelity HTS datasets, totalling over 28 million data points, with many targets possessing more than 1M labels. We then designed and evaluated machine learning models to assess the improvements offered by the integration of multi-fidelity data, including classical models and a bespoke, novel deep learning approach based on graph neural networks. Jointly modelling primary and confirmatory data led to a decrease of 12% in mean absolute error (MAE) and an increase of 152% in R-squared on the public datasets, and a reduction of 17% in MAE coupled with an uplift of 46% in R-squared on the AstraZeneca datasets (averaged across all evaluated methods). We conclude that joint modelling of multi-fidelity HTS data improves predictive performance and that deep learning enables the use of unique and highly desirable strategies such as leveraging signals from multi-million scale datasets and transfer learning.

show abstract

Section: Introductionmentioning

confidence: 99%

Multi-fidelity machine learning models for improved high-throughput screening predictions

Buterez

Janet

Kiddle

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Most production models are for absorption, distribution, metabolism, excretion, and toxicity (ADMET) end points because those are of interest to many therapeutic areas, and large numbers of diverse molecules (i.e., from many chemical classes) are assayed. For recent examples from other companies see Aleksić et al, Goller et al, and Cumming et al Merck & Co., Inc., Kenilworth, NJ, United States, has maintained multiple generations of infrastructure to generate and update QSAR models since 2005, most recently using QSAR Workbench as the platform. There is a Web-based interface on which chemists can sketch molecules and get predictions, but predictions may also be made on other platforms.…”

mentioning

confidence: 99%

“…Nowadays, it is recognized − that there are QSAR methods that give a slightly higher average predictivity. In practice, though, RF is still competitive because prediction accuracy for most data sets seems to be limited more by the data than the QSAR methodology. , In any case, our historical predictions are made with RF, so that is what we will examine here.…”

mentioning

confidence: 99%

“…This alone is not in accord with intuition; we might expect the predictivity to steadily increase with time because the models are presumably covering more chemical space with each version (from ∼20 000 to ∼56 000 molecules in the case of Papp). Aleksić et al had a similar observation at another pharma company. Can we explain why some time points are much worse than others?…”

mentioning

confidence: 99%

See 1 more Smart Citation

Prediction Accuracy of Production ADMET Models as a Function of Version: Activity Cliffs Rule

Sheridan

Culberson

Joshi

et al. 2022

J. Chem. Inf. Model.

View full text Add to dashboard Cite

As with many other institutions, our company maintains many quantitative structure-activity relationship (QSAR) models of absorption, distribution, metabolism, excretion, and toxicity (ADMET) end points and updates the models regularly. We recently examined version-to-version predictivity for these models over a period of 10 years. In this approach we monitor the goodness of prediction of new molecules relative to the training set of model version V before they are incorporated in the updated model V+1. Using a cell-based permeability assay (Papp) as an example, we illustrate how the QSAR models made from this data are generally predictive and can be utilized to enrich chemical designs and synthesis. Despite the obvious utility of these models, we turned up unexpected behavior in Papp and other ADMET activities for which the explanation is not obvious. One such behavior is that the apparent predictivity of the models as measured by root-mean-square-error can vary greatly from version to version and is sometimes very poor. One intuitively appealing explanation is that the observed activities of the new molecules fall outside the bulk of activities in the training set. Alternatively, one may think that the new molecules are exploring different regions of chemical space than the training set. However, the real explanation has to do with activity cliffs. If the observed activities of the new molecules are different than expected based on similar molecules in the training set, the predictions will be less accurate. This is true for all our ADMET end points.

show abstract

“…Typically a large number of diverse molecules are assayed for those end points. Aleksić et al and Goller et al discuss recent examples from pharma. Our company has been maintaining ADMET models since 2005 …”

mentioning

confidence: 99%

Stability of Prediction in Production ADMET Models as a Function of Version: Why and When Predictions Change

Sheridan

2022

J. Chem. Inf. Model.

View full text Add to dashboard Cite

As with other pharma companies, we maintain production QSAR models of ADMET end points and update them regularly. Here, for six ADMET end points, we examine the predictions of test set molecules on multiple versions of random forest models spanning a period of 10 years. For any given end point, the predictions for the majority of molecules are similar for all model versions. However, for a small minority of molecules, the prediction shifts substantially over the span of a few versions. For most molecules that shift, the prediction becomes more accurate at later times. This Perspective investigates metrics that can help indicate which molecules will shift substantially in prediction and when the shift will occur.

show abstract

ADMET Predictability at Boehringer Ingelheim: State‐of‐the‐Art, and Do Bigger Datasets or Algorithms Make a Difference?

Cited by 28 publications

References 48 publications

Multi-fidelity machine learning models for improved high-throughput screening predictions

Multi-fidelity machine learning models for improved high-throughput screening predictions

Prediction Accuracy of Production ADMET Models as a Function of Version: Activity Cliffs Rule

Stability of Prediction in Production ADMET Models as a Function of Version: Why and When Predictions Change

Contact Info

Product

Resources

About