Stability of Prediction in Production ADMET Models as a Function of Version: Why and When Predictions Change

Abstract

As with other pharma companies, we maintain production QSAR models of ADMET end points and update them regularly. Here, for six ADMET end points, we examine the predictions of test set molecules on multiple versions of random forest models spanning a period of 10 years. For any given end point, the predictions for the majority of molecules are similar for all model versions. However, for a small minority of molecules, the prediction shifts substantially over the span of a few versions. For most molecules that shift, the prediction becomes more accurate at later times. This Perspective investigates metrics that can help indicate which molecules will shift substantially in prediction and when the shift will occur.

Evaluation of Free Online ADMET Tools for Academic or Small Biotech Environments

“…During the construction of such a list, we have made some observations and experienced a series of drawbacks that we consider worthy of being mentioned: Websites change frequently because they are continuously being improved. Such mutability causes two types of problems: on the one hand, they are not always available to be used, and, on the other hand, the mathematical models can be modified, so the predictions can vary, as Sheridan pointed out in a recent paper [ 9 ]. In some cases, the web pages have simply disappeared during the preparation of this manuscript.…”

The (Re)-Evolution of Quantitative Structure–Activity Relationship (QSAR) Studies Propelled by the Surge of Machine Learning Methods

“…The second Perspective explores the reasons underlying the large changes in ADMET predictions for some molecules when different model versions were applied. 31 Interestingly, for most molecules with prediction changes, the prediction improved at later model versions. Metrics indicating which molecules will display large prediction changes were explored, leading to the observation that the molecules with large changes were associated with large prediction uncertainty in the models.…”

“…), grappling with significant issues for drug discovery, such as activity cliffs in the activity landscape of very similar molecules with very different activities, the representation of different conformations of small molecules, or how to share models without revealing the data of the training set, which can facilitate greater collaboration in the industry sector. Two Perspectives by Sheridan and co-workers 30,31 investigate the performance of multiple versions of RF models spanning a period of 10 years. Such models displayed unexpected behavior in the prediction of ADMET properties.…”

“…This could be explained by activity cliffs: the presence of molecules in the test set with activities different from similar molecules in the training set, leading to less accurate predictions. The second Perspective explores the reasons underlying the large changes in ADMET predictions for some molecules when different model versions were applied . Interestingly, for most molecules with prediction changes, the prediction improved at later model versions.…”

“…Two Perspectives by Sheridan and co-workers , investigate the performance of multiple versions of RF models spanning a period of 10 years. Such models displayed unexpected behavior in the prediction of ADMET properties.…”