Impact of Benchmark Data Set Topology on the Validation of Virtual Screening Methods: Exploration and Quantification by Spatial Statistics

Abstract

A common finding of many reports evaluating ligand-based virtual screening methods is that validation results vary considerably with changing benchmark data sets. It is widely assumed that these data set specific effects are caused by the redundancy, self-similarity, and cluster structure inherent to those data sets. These phenomena manifest themselves in the data sets' representation in descriptor space, which is termed the data set topology. A methodology for the characterization of data set topology based on spatial statistics is introduced. The method is nonparametric and can deal with arbitrary distributions of descriptor values. With this methodology it is possible to associate differences in virtual screening performance on different data sets with differences in data set topology. Moreover, the better virtual screening performance of certain descriptors can be explained by their ability of representing the benchmark data sets by a more favorable topology. Finally it is shown, that the composition of some benchmark data sets causes topologies that lead to overoptimistic validation results even in very "simple" descriptor spaces. Spatial statistics analysis as proposed here facilitates the detection of such biased data sets and may provide a tool for the future design of unbiased benchmark data sets.

Benchmarking Data Sets for the Evaluation of Virtual Ligand Screening Methods: Review and Perspectives

“…This choice is critical since the performances of a virtual screening method can vary considerably with the benchmarking dataset used for the study 74,75 . The first element that can guide the selection of a benchmarking dataset is the nature of the virtual screening method that will be evaluated.…”

Benchmarking methods and data sets for ligand enrichment assessment in virtual screening

“…The method to build MUV was the refined nearest neighbor analysis in spatial statistics [105]. First, 17 physicochemical properties were used for calculating pairwise Euclidean distances.…”

The effect of structural redundancy in validation sets on virtual screening performance

“…Research Article [8,12,13]. The lack of robustness that results from such bias can seriously skew retrospective analyses and mislead researchers as to which method is likely to give the best prospective performance.…”