OXBench: A benchmark for evaluation of protein multiple sequence alignment accuracy

Abstract

Background: The alignment of two or more protein sequences provides a powerful guide in the prediction of the protein structure and in identifying key functional residues, however, the utility of any prediction is completely dependent on the accuracy of the alignment. In this paper we describe a suite of reference alignments derived from the comparison of protein three-dimensional structures together with evaluation measures and software that allow automatically generated alignments to be benchmarked. We test the OXBench benchmark suite on alignments generated by the AMPS multiple alignment method, then apply the suite to compare eight different multiple alignment algorithms. The benchmark shows the current state-of-the art for alignment accuracy and provides a baseline against which new alignment algorithms may be judged.

Genome sequencing and next-generation sequence data analysis: A comprehensive compilation of bioinformatics tools and databases

“…Benchmark datasets like OXBench [37], HOMSTRAD [38], PREFAB [39], Bali-BASE [40], and SABmark [41] are built on real sequences by aligning structural elements and in some cases with hand-curation. Others like IRMBASE [42] are generated by simulating sequence evolution based on specific molecular evolutionary models.…”

Multiple sequence alignment for phylogenetic purposes

“…In addition, database searches are usually more effective when using amino acid sequences compared with nucleotide sequences, and this has provided impetus for other improvements. However, for nucleotide alignment, which is the prevalent mode in phylogenetic analyses, very little has changed in practice over the last 20 years (Taylor 1996;Phillips et al 2000;Raghava et al 2003), when the progressive-alignment strategy was first developed (Hogeweg and Hesper 1984;Feng and Doolittle 1987;Taylor 1987). Nevertheless, some of these developments could usefully be moved across to phylogenetic analysis of nucleotides as well, and they are discussed in more detail in several sections below.…”

“…Kjer 1995Kjer , 2004Hickson et al 1996;Jennings et al 2001;Simossis and Heringa 2004). As an example of the perceived importance of this relationship, evaluation of the quality of procedures for multiple sequence alignment is now frequently assessed using structure-based reference alignments as the best estimate of the 'correct' alignment, including BAliBASE (Thompson et al 1999a(Thompson et al , 2005Bahr et al 2001), OXBench (Raghava et al 2003), PREFAB (Edgar 2004b), SABmark (Van Walle et al 2005), IRMBase (Subramanian et al 2005) and BRAliBASE (Gardner et al 2005); and a similar approach has been taken to evaluation of pairwise alignment for database searches and structure prediction (Brenner et al 1998;Domingues et al 2000;Sauder et al 2000;Marsden and Abagyan 2004;Marti-Renom et al 2004). Note that this approach explicitly uses a biological criterion to judge the quality of the alignments, rather than whatever mathematical criterion was used to produce the alignments.…”

“…There is now much empirical evidence that multiplesequence alignments produced by the pattern-matching programs are not necessarily similar to those based on structure or function considerations, both for amino acids (Taylor 1986;Barton and Sternberg 1987;Johnson et al 1990;Gotoh 1996;Briffeuil et al 1998;Thompson et al 1999b;Katoh et al 2002Katoh et al , 2005aKatoh et al , 2005bLassmann and Sonnhammer 2002;Marchler-Bauer et al 2002;Raghava et al 2003;Edgar 2004b;Van Walle et al 2004;Yamada et al 2004;Do et al 2005;Subramanian et al 2005;Zhou and Zhou 2005;Sze et al 2006) and for nucleotides (Ellis and Morrison 1995;Kjer 1995;Morrison and Ellis 1997;Beebe et al 2000;Hickson et al 2000;Mugridge et al 2000;Gardner et al 2005;Katoh et al 2005b;Lebrun et al 2006). It is therefore worthwhile to explore this problem in more detail.…”

“…Thus, various improvements to the simple progressive alignment algorithm have been developed, designed to deal with the fundamental limitation, which is that misalignments made early in the progressive process are not subsequently corrected. Several the associated computer programs have been compared for their ability to detect specified conserved sequence motifs (McClure et al 1994;Briffeuil et al 1998;Hudak and McClure 1999;Hickson et al 2000) and for the overall match of their alignments to structurebased alignments (Gotoh 1996;Morrison and Ellis 1997;Thompson et al 1999b;Notredame et al 2000;Karplus and Hu 2001;Katoh et al 2002Katoh et al , 2005aKatoh et al , 2005bLassmann and Sonnhammer 2002;Raghava et al 2003;Edgar 2004b;Grasso and Lee 2004;Van Walle et al 2004;Do et al 2005;Subramanian et al 2005;Zhou and Zhou 2005;Sze et al 2006) or to simulated alignments (Pollard et al 2004;Ogden and Rosenberg 2006). None of the programs are completely successful based on any of the three criteria, and their success rate can sometimes be very low.…”