Optimization of CAMD techniques 3. Virtual screening enrichment studies: a help or hindrance in tool selection?

Abstract

Over the last few years many articles have been published in an attempt to provide performance benchmarks for virtual screening tools. While this research has imparted useful insights, the myriad variables controlling said studies place significant limits on results interpretability. Here we investigate the effects of these variables, including analysis of calculation setup variation, the effect of target choice, active/decoy set selection (with particular emphasis on the effect of analogue bias) and enrichment data interpretation. In addition the optimization of the publicly available DUD benchmark sets through analogue bias removal is discussed, as is their augmentation through the addition of large diverse data sets collated using WOMBAT.

Recommendations for evaluation of computational methods

“…Many docking approaches require all atoms, and nearly all require at least the positions of the polar protons. Without the precise protein structures used, in a widely used file format, it is not possible to reproduce the results of a report or make comparisons of other methods to those reported [7,9,11] [4,[7][8][9].…”

“…b. All the actives are chemically similar [2,4,5,8,10]. This is more relevant to ligand-based methods, but also applicable to docking because operationally finding chemically similar molecules as being potentially active is of little value in that these will likely be found by other methods.…”

“…Protein structure selection should take into account more than just the nominal resolution [4,5,9]. There are other measures such as coordinate precision that are more appropriate but require the use of structures where an R and R free are reported.…”

“…c. The most relevant tests of methods will employ proteins whose structure was determined with a ligand other than the one being predicted or a close analog thereof [8]. d. The number and diversity of protein targets needs to be sufficient to enable to draw statistically robust conclusions [4,6,10,11]. Some typical targets (e.g.…”

“…This special issue of the Journal of Computer-Aided Molecular Design includes eleven papers, each of which makes a detailed study of at least one aspect of methodological evaluation [1][2][3][4][5][6][7][8][9][10][11]. The papers collected within this issue make the detailed case for the recommendations that follow; the recommendations are intended to provide guidance to editorial boards and reviewers of work submitted for publication in our field.…”