Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction

Abstract

RNA splicing is an important post-transcriptional process of gene expression in eukaryotic organisms. Here, we developed a novel language model, SpliceBERT, pre-trained on the precursor messenger RNA sequences of 72 vertebrates to improve sequence-based modelling of RNA splicing. SpliceBERT is capable of generating embeddings that preserve the evolutionary information of nucleotides and functional characteristics of splice sites. Moreover, the pre-trained model can be utilized to prioritize potential splice-disrupting variants in an unsupervised manner based on genetic variants' impact on the output of SpliceBERT for sequence context. Benchmarked on a multi-species splice site and a human branchpoint prediction task, SpliceBERT outperformed not only conventional baseline models but also other language models pretrained only on the human genome. Our study highlighted the importance of unsupervised learning with genomic sequences of multiple species and indicated that language models were promising approaches to decipher the determinants of RNA splicing.

Evaluating the representational power of pre-trained DNA language models for regulatory genomics

“…Notably, we elected to not fine-tune weights of the gLM on each downstream task, which is how gLMs have been previously benchmarked 23,24,26,30,40 . While gLM performance would likely improve with fine-tuning, the scope of this study was to strictly gauge the knowledge of cis -regulatory biology learned during pre-training.…”

“…The utility of gLMs pre-trained on whole genomes for studying the non-coding genome has been limited. Previous benchmarks have largely considered gLMs that have been fine-tuned (i.e., adjusting the weights of the gLM) on each downstream task 23,24,26,30,40 . In each benchmark, a fine-tuned gLM has demonstrated improved predictions on a host of downstream prediction tasks, often based on the classification of functional elements, such as histone marks or promoter annotations.…”

“…Recently, there has been a surge of pre-trained gLMs [20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][38][39][40][41][42][43] . gLMs take as input DNA sequences that have undergone tokenization, an encoding scheme applied to either a single nucleotide or k-mer of nucleotides.…”

“…Alternatively, the base architecture has also been constructed with a stack of residual-connected convolution blocks, either with dilated convolutional layers 20 or implicit convolutions with a Hyena operator 21,22,49 . The pre-training data can vary significantly, encompassing the whole genome of a single species 20,24,32 or the whole genomes across multiple species 23,25,26,28,33 or focused only within specific regions of the genomes, such as the untranslated regions (UTRs) 29 , pre-mRNA 30 , promoters 22 , coding regions [35][36][37] , non-coding RNA 40 , or conserved sites 34 .…”

Perturbation-aware predictive modeling of RNA splicing using bidirectional transformers

“…Existing transformer models for predicting RNA splicing from sequence (e.g. SpliceBERT 8 ) have generally been trained on transcriptomic reference databases such as RefSeq or GENCODE, thus ignoring the large amount of information encoded by cell-type specific RNA processing patterns. Additionally, long-read sequencing has emerged as the gold standard in accurately capturing the complexity of RNA splicing events, yet the current datasets derived from long-read sequencing are relatively small and primarily observational.…”