Deep learning models for RNA secondary structure prediction (probably) do not generalize across families

Abstract

Motivation The secondary structure of RNA is of importance to its function. Over the last few years, several papers attempted to use machine learning to improve de novo RNA secondary structure prediction. Many of these papers report impressive results for intra-family predictions, but seldom address the much more difficult (and practical) inter-family problem. Results We demonstrate that it is nearly trivial with convolutional neural networks to generate pseudo-free energy changes, modeled after structure mapping data, that improve the accuracy of structure prediction for intra-family cases. We propose a more rigorous method for inter-family cross-validation that can be used to assess the performance of learning-based models. Using this method, we further demonstrate that intra-family performance is insufficient proof of generalisation despite the widespread assumption in the literature, and provide strong evidence that many existing learning-based models have not generalised inter-family. Availability Source code and data is available at https://github.com/marcellszi/dl-rna. Supplementary information Supplementary data are available at Bioinformatics online.

The Master Database of All Possible RNA Sequences and Its Integration with RNAcmap for RNA Homology Search

“…SPOT-RNA [36], MXfold2 [37], UFold [38]). However, there is a risk of overtraining for some of these deep learning techniques, which would make some methods to perform poorly for unseen RNA families [39]. Thus, caution must be exercised when using these deep learning techniques.…”

Advances in RNA 3D Structure Prediction

“…Recently, more and more ML-based methods of RNA secondary structure prediction have been proposed . This is because the number of RNAs with known secondary structures is much larger than that of known 3D structures.…”

“…It even has had a profound impact on the field of structural biology. Recently, different ML models have also been applied to solve 73 This is because the number of RNAs with known secondary structures is much larger than that of known 3D structures. For example, the RNA secondary structure database bpRNA-1m has 102,348 RNA sequences.…”

Sequence similarity governs generalizability of de novo deep learning models for RNA secondary structure prediction

“…A number of highly successful de novo DL models have been reported, such as 2dRNA [21], ATTfold [22], DMfold [23], E2Efold [24], MXfold2 [25], SPOT-RNA [26], and Ufold [27], among others [28][29][30][31]. These DL models markedly outperform traditional algorithms, with even close-to-perfect predictions in some cases, though questions on the training vs. test similarity have been raised [32,33] and discussed below. It is worth noting that DL models have also been developed for base-level prediction tasks such as the pairing probability of each base [34].…”