Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization

Abstract

Undetected overfitting can occur when there are significant redundancies between training and validation data. We describe AVE, a new measure of training-validation redundancy for ligand-based classification problems, that accounts for the similarity among inactive molecules as well as active ones. We investigated seven widely used benchmarks for virtual screening and classification, and we show that the amount of AVE bias strongly correlates with the performance of ligand-based predictive methods irrespective of the predicted property, chemical fingerprint, similarity measure, or previously applied unbiasing techniques. Therefore, it may be the case that the previously reported performance of most ligand-based methods can be explained by overfitting to benchmarks rather than good prospective accuracy.

Machine learning classification can reduce false positives in structure-based virtual screening

“…The DEKOIS project (currently at version 2.0) [71,72] is intended to provide a "demanding" Among the 81 proteins in the DEKOIS set, we noted that some were included in our training set as well. To avoid any potential information leakage that might overestimate the performance we could expect in future applications [33], we completely removed these testcases. This left a set of 23 protein targets, each of which vScreenML had never seen before.…”

“…For these reasons, machine learning techniques may be especially well-suited for developing scoring functions that will provide a dramatic improvement in the ability to identify active compounds without human expert intervention. However, while machine learning may offer the potential to improve on the high false positive rate of current scoring function, further analysis has revealed that many methods to date reporting promising results in artificial benchmark experiments may have inadvertently overfit models to the training data [33]: this can be a subtle effect of information leakage, occurring when the validation/testing data are not truly non-redundant from the training data.…”

Dataset Augmentation Allows Deep Learning-Based Virtual Screening To Better Generalize To Unseen Target Classes, And Highlight Important Binding Interactions

“…This version of the CNN 16 was the subject of all the analyses so far carried out into the importance of the receptor in active/decoy classification. [1][2][3] DenseFS: A much deeper network with three sets of four densely connected convolutional layers followed by a fully-connected softmax layer and cross entropy loss. 18 This network significantly improved performance over the Gnina network on both held-out DUD-E targets and the ChEMBL set.…”

“…A series of recent papers has shown that some deep learning methods designed for structurebased virtual screening can accurately separate actives and decoys when given only the structure of the ligand. [1][2][3] These results indicate that such methods are learning differences between the properties of actives and decoys, rather than the physical interactions between the receptor and the ligand. From this it is possible to conclude both that the methods will fail to generalize well (predict on datasets far removed from the training data), and that there are significant flaws in the current training datasets and/or regimens.…”

“…It was recently shown that model performance for some methods does not suffer significantly when ligand structures in the test set are given without the protein target/receptor. 2,3 The overall conclusion from these studies was that the methods are using very little if any information from the protein target. This finding links to the fact that in many of these datatsets, receptor-free methods such as k-nearest neighbours on ligand fingerprints perform almost as well as methods making use of the receptor.…”