CompaRNA: a server for continuous benchmarking of automated methods for RNA secondary structure prediction

Abstract

We present a continuous benchmarking approach for the assessment of RNA secondary structure prediction methods implemented in the CompaRNA web server. As of 3 October 2012, the performance of 28 single-sequence and 13 comparative methods has been evaluated on RNA sequences/structures released weekly by the Protein Data Bank. We also provide a static benchmark generated on RNA 2D structures derived from the RNAstrand database. Benchmarks on both data sets offer insight into the relative performance of RNA secondary structure prediction methods on RNAs of different size and with respect to different types of structure. According to our tests, on the average, the most accurate predictions obtained by a comparative approach are generated by CentroidAlifold, MXScarna, RNAalifold and TurboFold. On the average, the most accurate predictions obtained by single-sequence analyses are generated by CentroidFold, ContextFold and IPknot. The best comparative methods typically outperform the best single-sequence methods if an alignment of homologous RNA sequences is available. This article presents the results of our benchmarks as of 3 October 2012, whereas the rankings presented online are continuously updated. We will gladly include new prediction methods and new measures of accuracy in the new editions of CompaRNA benchmarks.

Genome-wide discovery of miRNAs using ensembles of machine learning algorithms and logistic regression

“…ContextFold clearly outperforms the other methods, achieving an MCC of 0.80 and 0.56, respectively. The similar performance obtained for the energy-based methods (RNAfold, UNAfold and Sfold) and the higher performance of ContextFold is consistent with previous research (Gardner and Giegerich, 2004;Puton et al, 2013). For both datasets, all methods obtained a higher median specificity than sensitivity ( Table 3), indicating that it is more difficult to predict the true pairs than to not predict the false ones.…”

“…After all, intergenic pre-miRNAs are found on transcripts with lengths ranging from a few hundred to several thousand nucleotides (Saini et al, 2007). Furthermore, the MFE structure is not guaranteed to be the true biological structure of the transcript (Puton et al, 2013). These concerns motivate closer inspection of the secondary structure prediction step for in silico prediction of premiRNAs.…”

“…Comprehensive and up-to-date benchmarks can be found at the CompaRNA website (Puton et al, 2013). The purpose here is to investigate more closely the performance of some structure prediction software either used by pre-miRNA discovery tools or performing well in the CompaRNA benchmark.…”

“…To count the number of True Positives (TP), False Negatives (FN), True Negatives (TN) and False Positives (FP) the convention used by Gardner and Giegerich (2004) and CompaRNA (Puton et al, 2013) was applied. TP denotes predicted base pairs that also exist in the true structure; FN denotes true base pairs not in the predicted structure; TN denotes base pairs not in the true structure that are not predicted; FP denotes base pairs that are predicted but do not exist in the true structure.…”

RNA structure prediction: Progress and perspective

“…[9,18] Since the incipient manual comparative sequence analysis methods highly depend on the knowledge of users and are time-consuming, several automatic algorithms [19][20][21][22][23][24][25] that use multiple sequences were recently developed to predict RNA secondary structures , such as RNAalifold [20] , Pfold/PPfold [21,22] , TurboFold [23] , RNAforester [24] and MARNA [25] . These algorithms have been benchmarked for prediction speed and accuracy with the same data set [26] and have been reviewed elsewhere [16] . Here, we only give a brief overview.…”