Is it better to combine predictions?

Abstract

We have compared the accuracy of the individual protein secondary structure prediction methods: PHD, DSC, NNSSP and Predator against the accuracy obtained by combing the predictions of the methods. A range of ways of combing predictions were tested: voting, biased voting, linear discrimination, neural networks and decision trees. The combined methods that involve 'learning' (the non-voting methods) were trained using a set of 496 non-homologous domains; this dataset was biased as some of the secondary structure prediction methods had used them for training. We used two independent test sets to compare predictions: the first consisted of 17 non-homologous domains from CASP3 (Third Community Wide Experiment on the Critical Assessment of Techniques for Protein Structure Prediction); the second set consisted of 405 domains that were selected in the same way as the training set, and were non-homologous to each other and the training set. On both test datasets the most accurate individual method was NNSSP, then PHD, DSC and the least accurate was Predator; however, it was not possible to conclusively show a significant difference between the individual methods. Comparing the accuracy of the single methods with that obtained by combing predictions it was found that it was better to use a combination of predictions. On both test datasets it was possible to obtain a approximately 3% improvement in accuracy by combing predictions. In most cases the combined methods were statistically significantly better (at P = 0.05 on the CASP3 test set, and P = 0.01 on the EBI test set). On the CASP3 test dataset there was no significant difference in accuracy between any of the combined method of prediction: on the EBI test dataset, linear discrimination and neural networks significantly outperformed voting techniques. We conclude that it is better to combine predictions.

Modular Organization of Rabies Virus Phosphoprotein

“…43,51 Numerous algorithms, available through WEB servers, predict the location of secondary structure elements on the basis of the amino acid sequence, and some algorithms combine multiple methods. 52,53 Here again, we used a simple voting method for combining predictions from different algorithms and for defining consensus locations of predicted secondary structures (SS-score).…”

Genomic Computing. Explanatory Analysis of Plant Expression Profiling Data Using Machine Learning

“…Another feature of GPs is that because of the stochastic way in which they are initiated and evolve, they are not deterministic. This said, it is possible to turn such properties to advantage by running the GP several times, as there is plenty of evidence that combining or voting among several independent solutions to a problem can give improved learning (Drucker et al, 1994;Bauer and Kohavi, 1999;Dietterich, 2000aDietterich, , 2000bFriedman et al, 2000;King et al, 2000). Finally, all computerintensive methods of this type, including those based on purely multivariate statistical strategies (Martens and Naes, 1989), have a great many degrees of freedom, which require that we provide a careful evaluation with respect to the reality of the solutions found (Chatfield, 1995).…”

Ensemble of Neural Networks to Solve Class Imbalance Problem of Protein Secondary Structure Prediction

“…Further improvement in preformance of PSSP were also acheived by exploiting evolutioary information via multiple sequence alignments (MSAs) profiles [10], position-specific score matrices (PSSM) [15], homology detection using hidden markov models [16] and PSI-BLAST [17]. Other approaches have been applied to PSSP are support vector machines (SVM) [18], [19] and ensemble methods that combine some machine learning approaches [20] via the majority voting or weighted majority voting techniques. These methods could optain up to a 3% improvement in Q 3 accuracy over the best individual method.…”