CSAR Benchmark Exercise 2013: Evaluation of Results from a Combined Computational Protein Design, Docking, and Scoring/Ranking Challenge

Smith, Richard; Damm-Ganamet, Kelly L.; Dunbar, James B.; Ahmed, Aqeel; Chinnaswamy, Krishnapriya; Delproposto, James; Kubish, Ginger; Tinberg, Christine E.; Khare, Sagar D.; Dou, Jiayi; Doyle, Lindsey; Stuckey, Jeanne A.; Baker, David; Carlson, Heather A.

doi:10.1021/acs.jcim.5b00387

Cited by 55 publications

(47 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We participated in the 2013/14 CSAR challenge that involved rank-ordering compounds to homology models of the receptors with a given protein primary sequence, identifying close-to-native bound conformations out of a set of decoy poses, and rank-ordering the affinity of sets of congeneric compounds to a given protein. Our predictions were among the best in the field [29, 30]. We showed that the most significant contribution to a meaningful enrichment of native-like models was the identification of the best receptor structure for docking and scoring.…”

Section: Introductionmentioning

confidence: 79%

Optimal strategies for virtual screening of induced-fit and flexible target in the 2015 D3R Grand Challenge

Baumgartner

Wingert

et al. 2016

J Comput Aided Mol Des

View full text Add to dashboard Cite

Induced fit or protein flexibility can make a given structure less useful for docking and/or scoring. The 2015 Drug Design Data Resource (D3R) Grand Challenge provided a unique opportunity to prospectively test optimal strategies for virtual screening in these type of targets: heat shock protein 90 (HSP90), a protein with multiple ligand-induced binding modes; and, mitogen-activated protein kinase kinase kinase kinase 4 (MAP4K4), a kinase with a large flexible pocket. Using previously known co-crystal structures, we tested predictions from methods that keep the receptor structure fixed and used (a) multiple receptor/ligand co-crystals as binding templates for minimization or docking (“close”), (b) methods that align or dock to a single receptor (“cross”), and (c) a hybrid approach that chose from multiple bound ligands as initial templates for minimization to a single receptor (“min-cross”). Pose prediction using our “close” models resulted in average ligand RMSDs of 0.32 Å and 1.6 Å for HSP90 and MAP4K4, respectively, the most accurate models of the community-wide challenge. On the other hand, affinity ranking using our “cross” methods performed well overall despite the fact that a fixed receptor cannot model ligand-induced structural changes,. In addition, “close” methods that leverage the co-crystals of the different binding modes of HSP90 also predicted the best affinity ranking. Our studies suggest that analysis of changes on the receptor structure upon ligand binding can help select an optimal virtual screening strategy.

show abstract

Section: Introductionmentioning

confidence: 79%

Optimal strategies for virtual screening of induced-fit and flexible target in the 2015 D3R Grand Challenge

Baumgartner

Wingert

et al. 2016

J Comput Aided Mol Des

View full text Add to dashboard Cite

show abstract

“…These datasets were primarily obtained from the pharmaceutical industry such as Abbvie 28 , Vertex 28 , and GlaxoSmithKline (GSK). A few datasets were obtained from academia, such as one from the Baker group that formed our 2013 exercise 27 . Part of the remit of the CSAR Center was to run regular exercises to actively engage the docking and scoring community in assessing the current state of the art and the impact of potential improvements.…”

Section: Introductionmentioning

confidence: 99%

CSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma

Carlson

Smith

Damm-Ganamet

et al. 2016

J. Chem. Inf. Model.

Self Cite

View full text Add to dashboard Cite

The 2014 CSAR Benchmark Exercise was the last community-wide exercise that was conducted by the group at the University of Michigan, Ann Arbor. For this event, GlaxoSmithKline (GSK) donated unpublished crystal structures and affinity data from in-house projects. Three targets were used: tRNA (m1G37) methyltransferase (TrmD), Spleen Tyrosine Kinase (SYK), and Factor Xa (FXa). A particularly strong feature of the GSK data is its large size, which lends greater statistical significance to comparisons between different methods. In Phase 1 of the CSAR 2014 Exercise, participants were given several protein-ligand complexes and asked to identify the one near-native pose from among 200 decoys provided by CSAR. Though decoys were requested by the community, we found that they complicated our analysis. We could not discern whether poor predictions were failures of the chosen method or an incompatibility between the participant’s method and the setup protocol we used. This problem is inherent to decoys and we strongly advise against their use. In Phase 2, participants had to dock and rank/score a set of small molecules given only the SMILES strings of the ligands and a protein structure with a different ligand bound. Overall, docking was a success for most participants, much better in Phase 2 than in Phase 1. However, scoring was a greater challenge. No particular approach to docking and scoring had an edge, and successful methods included empirical, knowledge-based, machine-learning, shape-fitting, and even those with solvation and entropy terms. Several groups were successful in ranking TrmD and/or SYK, but ranking FXa ligands was intractable for all participants. Methods that were able to dock well across all submitted systems include MDock1, Glide-XP2, PLANTS3, Wilma4, Gold5, SMINA6, Glide-XP2/PELE7, FlexX8, and MedusaDock9. In fact, the submission based on Glide-XP2/PELE7 cross-docked all ligands to many crystal structures, and it was particularly impressive to see success across an ensemble of protein structures for multiple targets. For scoring/ranking, submissions that showed statistically significant achievement include MDock1 using ITScore1,10 with a flexible-ligand term11, SMINA6 using Autodock-Vina12,13, FlexX8 using HYDE14, and Glide-XP2 using XP DockScore2 with and without ROCS15 shape similarity16. Of course, these results are for only three protein targets, and many more systems need to be investigated to truly identify which approaches are more successful than others. Furthermore, our exercise is not a competition.

show abstract

“…According to the CSAR organizer’s paper 42 , our results using Vina is the 6 th among 27 predictions submitted to this phase, a larger correlation (0.819 and 0.834), while the results with the combined score and PL-PatchSurfer were among the lower ranks.…”

Section: Resultsmentioning

confidence: 93%

Combined Approach of Patch-Surfer and PL-PatchSurfer for Protein–Ligand Binding Prediction in CSAR 2013 and 2014

Zhu

Shin

Kim

2015

J. Chem. Inf. Model.

View full text Add to dashboard Cite

The Community Structure-Activity Resource (CSAR) benchmark exercise provides a unique opportunity for researchers to objectively evaluate the performance of protein-ligand docking methods. Patch-Surfer and PL-PatchSurfer, molecular surface-based methods for predicting binding ligands of proteins developed in our group, were tested on both CSAR 2013 and 2014 benchmark exercises in combination with an empirical scoring function-based method, AutoDock, while we only participated in CSAR 2013 using Patch-Surfer. The prediction results for Phase 1 task in CSAR 2013 showed that Patch-Surfer was able to rank all the four designed binding proteins within top ranks, outperforming AutoDock Vina. In Phase 2 of 2013, PL-PatchSurfer correctly selected the correct ligand pose for two target proteins. PL-PatchSurfer performed reasonably well in ranking ligands according to their binding affinity and in selecting near-native ligand poses in 2013 Phase 3 and 2014 Phase 1, respectively, although AutoDock Vina showed better performance. Lastly, in the 2014 Phase 2 exercise, the PL-PatchSurfer scores computed for ligands to target protein pairs correlated well with their pIC50 values, which was better or comparable to results by other participants. Overall, our methods showed fairly good performance in CSAR 2013 and 2014. Unique characteristics of the methods are discussed in comparison with AutoDock.

show abstract

CSAR Benchmark Exercise 2013: Evaluation of Results from a Combined Computational Protein Design, Docking, and Scoring/Ranking Challenge

Cited by 55 publications

References 60 publications

Optimal strategies for virtual screening of induced-fit and flexible target in the 2015 D3R Grand Challenge

Optimal strategies for virtual screening of induced-fit and flexible target in the 2015 D3R Grand Challenge

CSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma

Combined Approach of Patch-Surfer and PL-PatchSurfer for Protein–Ligand Binding Prediction in CSAR 2013 and 2014

Contact Info

Product

Resources

About