Evaluation of predictions in the CASP10 model refinement category

Abstract

Here we report on the assessment results of the third experiment to evaluate the state of the art in protein model refinement, where participants were invited to improve the accuracy of initial protein models for 27 targets. Using an array of complementary evaluation measures, we find that five groups performed better than the naïve (null) method—a marked improvement over CASP9, although only three were significantly better. The leading groups also demonstrated the ability to consistently improve both backbone and side chain positioning, while other groups reliably enhanced other aspects of protein physicality. The top-ranked group succeeded in improving the backbone conformation in almost 90% of targets, suggesting a strategy that for the first time in CASP refinement is successful in a clear majority of cases. A number of issues remain unsolved: the majority of groups still fail to improve the quality of the starting models; even successful groups are only able to make modest improvements; and no prediction is more similar to the native structure than to the starting model. Successful refinement attempts also often go unrecognized, as suggested by the relatively larger improvements when predictions not submitted as model 1 are also considered. Proteins 2014; 82(Suppl 2):98–111.

Protein structure refinement by optimization

“…The predictors in CASP are also tested for their ability to refine models [6]. The purpose of this category is to develop methods that can draw models closer to the native structure and thereby improve the accuracy of the models.…”

“…In the most simple case the basis functions are contact potentials which are turned on when the distance between two amino acids is within an interval. Explicitly, the potential is defined as 6) where r i,j is the distance between the i-th and j-th interaction pair and…”

“…Knowledgebased potentials are trained with the purpose of improving the quality of near-native structures. This means either to rank a set of near-native protein structures according to how native-like they are (model quality assessment [5]) or to refine the quality of near-native protein structures (protein structure refinement [6]). Most knowledgebased potentials are stochastic but in this study a knowledge-based potential is a model of the energy landscape of proteins that is trained on a set of near-native structures called decoys.…”

“…The predicted structures do not always achieve sufficiently high accuracy for them to be used for applications such as drug design, drug screening, ligand docking or molecular replacement [4]. With the purpose of improving the accuracy of protein structures, every second year several groups participate in the model refinement category in the Critical Assessment of protein Structure Prediction (CASP) [6]. The number of participating research groups has increased remarkedly from CASP 7 to CASP 10 and, as a consequence, a number of new methods has been developed.…”

“…The number of participating research groups has increased remarkedly from CASP 7 to CASP 10 and, as a consequence, a number of new methods has been developed. Despite a growing interest, the improvements in the refinement targets have been small and the methods have not shown the consistency necessary for them to be used in practise [6]. A consistent method to improve the accuracy of protein structures is, therefore, desirable and this work is a step in that direction.…”