CAFASP‐1: Critical assessment of fully automated structure prediction methods

Abstract

The results of the first Critical Assessment of Fully Automated Structure Prediction (CAFASP-1) are presented. The objective was to evaluate the success rates of fully automatic web servers for fold recognition which are available to the community. This study was based on the targets used in the third meeting on the Critical Assessment of Techniques for Protein Structure Prediction (CASP-3). However, unlike CASP-3, the study was not a blind trial, as it was held after the structures of the targets were known. The aim was to assess the performance of methods without the user intervention that several groups used in their CASP-3 submissions. Although it is clear that "human plus machine" predictions are superior to automated ones, this CAFASP-1 experiment is extremely valuable for users of our methods; it provides an indication of the performance of the methods alone, and not of the "human plus machine" performance assessed in CASP. This information may aid users in choosing which programs they wish to use and in evaluating the reliability of the programs when applied to their specific prediction targets. In addition, evaluation of fully automated methods is particularly important to assess their applicability at genomic scales. For each target, groups submitted the top-ranking folds generated from their servers. In CAFASP-1 we concentrated on fold-recognition web servers only and evaluated only recognition of the correct fold, and not, as in CASP-3, alignment accuracy. Although some performance differences appeared within each of the four target categories used here, overall, no single server has proved markedly superior to the others. The results showed that current fully automated fold recognition servers can often identify remote similarities when pairwise sequence search methods fail. Nevertheless, in only a few cases outside the family-level targets has the score of the top-ranking fold been significant enough to allow for a confident fully automated prediction. Because the goals, rules, and procedures of CAFASP-1 were different from those used at CASP-3, the results reported here are not comparable with those reported in CASP-3. Nevertheless, it is clear that current automated fold recognition methods can not yet compete with "human-expert plus machine" predictions. Finally, CAFASP-1 has been useful in identifying the requirements for a future blind trial of automated served-based protein structure prediction.

New HEAT-like repeat motifs in proteins regulating proteasome structure and function

“…The REP server (http://www.embl-heidelberg.de/ andrade/papers/rep/search.html) containing more specific sequence profiles of ARM/HEAT repeats (Andrade et al, 2001) and protein structure prediction servers SAM-T99, 3D-PSSM, and PSIPRED (Fischer et al, 1999;Karplus et al, 1997;McGuffin et al, 2000) produced essentially the same results. Since ARM/HEAT repeats are often found in extended tracts of tandem repeats (Andrade et al, 2001;Kobe and Kajava, 2000) we probed for additional copies using the generalized sequence profile method (Bucher et al, 1996).…”

DOMpro: Protein Domain Prediction Using Profiles, Secondary Structure, Relative Solvent Accessibility, and Recursive Neural Networks

“…DOMpro was independently evaluated along with 12 other predictors in the Critical Assessment of Fully Automated Structure Prediction 4 (CAFASP-4) (Fischer et al, 1999;Saini and Fischer, 2005). The results, kindly provided by Dr. Saini, are available at http://cafasp4.bioinformatics.buffalo.edu/dp/update.html.…”

CASP: A Driving Force in Protein Structure Modeling

“…Server performance has been continuously checked by CASP starting in CASP3 (originally with the help of the Critical Assessment of Fully Automated Structure Prediction Methods [CAFASP] [38] and since CASP6, independently). Analysis of server performance in successive CASPs [28,39] shows that the best human -expert groups in CASP still outperform the best server groups, but the gap between the best servers and the best human -expert groups is narrowing.…”