Uni-Rna: Universal Pre-Trained Models Revolutionize Rna Research

Abstract

RNA molecules play a crucial role as intermediaries in diverse biological processes. Attaining a profound understanding of their function can substantially enhance our comprehension of life’s activities and facilitate drug development for numerous diseases. The advent of high-throughput sequencing technologies makes vast amounts of RNA sequence data accessible, which contains invaluable information and knowledge. However, deriving insights for further application from such an immense volume of data poses a significant challenge. Fortunately, recent advancements in pre-trained models have surfaced as a revolutionary solution for addressing such challenges owing to their exceptional ability to automatically mine and extract hidden knowledge from massive datasets. Inspired by the past successes, we developed a novel context-aware deep learning model named Uni-RNA that performs pre-training on the largest dataset of RNA sequences at the unprecedented scale to date. During this process, our model autonomously unraveled the obscured evolutionary and structural information embedded within the RNA sequences. As a result, through fine-tuning, our model achieved the state-of-the-art (SOTA) performances in a spectrum of downstream tasks, including both structural and functional predictions. Overall, Uni-RNA established a new research paradigm empowered by the large pre-trained model in the field of RNA, enabling the community to unlock the power of AI at a whole new level to significantly expedite the pace of research and foster groundbreaking discoveries.

ERNIE-RNA: An RNA Language Model with Structure-enhanced Representations

“…Compared to UFold 22 that is considered as the current state-of-the-art end-to-end method for RNA secondary structure prediction, ERNIE-RNA improves the average binary f1 score by 14.4% to 0.748. Despite ERNIE-RNA having less than one-fifth of the model size and pretrained with less than one-fiftieth of the data size compared to UNI-RNA 32 , ERNIE-RNA outperforms UNI-RNA, the currently largest pre-trained RNA language model, further boosting the average macro-average f1 score by 6.3% to 0.873. We further compared ERNIR-RNA’s prediction performance to RNA-FM at the single sample level.…”

“…Nowadays, the advancements in high throughput sequencing technology 26 have produced a wealth of unlabeled data, which contain rich information about RNA structures and functions. Many BERT-style 27 RNA language models trained on abundant RNA sequences have been reported, such as RNA-FM 28 , RNABERT 29 , RNA-MSM 30 , CodonBERT 31 and UNI-RNA 32 . RNA-FM is a BERT-based RNA foundation model trained on 23 million unannotated RNA sequences, demonstrating applications in predicting both structural and functional properties.…”

RNA-TorsionBERT: leveraging language models for RNA 3D torsion angles prediction

“…RNABERT was trained on 76,237 human-derived small ncRNAs. Other methods have been adapted to RNA language but uses MSA as inputs like RNA-FM (29), Uni-RNA (30) or RNA-MSM (31). Nonetheless, they require multiple sequence alignment (MSA) as inputs, which restricts the use for RNAs.…”

“…For the proteins, EMSFold (16) has built a successful language model (without the use of MSA) for protein 3D structure prediction, which achieves competitive results. For RNA, methods have been developed to leverage language-based approaches for RNA structural predictions, like RNA-FM (17), Uni-RNA (18) or RNA-MSM (19). Nonetheless, they require multiple sequence alignment (MSA) as inputs, which restricts the use for RNAs.…”

ATOM-1: A Foundation Model for RNA Structure and Function Built on Chemical Mapping Data

“…Additionally, and in contrast to MSAs, chemical mapping experiments can be run on arbitrary RNA sequences, allowing the exploration of sequence space beyond natural sequences. As these experiments directly measure structural information, foundation models trained on chemical mapping data could enable better predictions on structure-related tasks for RNAs of interest compared to models trained on natural sequences alone [22][23][24].…”