The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

Abstract

Closing the gap between measurable genetic information and observable traits is a longstanding challenge in genomics. Yet, the prediction of molecular phenotypes from DNA sequences alone remains limited and inaccurate, often driven by the scarcity of annotated data and the inability to transfer learnings between prediction tasks. Here, we present an extensive study of foundation models pre-trained on DNA sequences, named the Nucleotide Transformer, integrating information from 3,202 diverse human genomes, as well as 850 genomes from a wide range of species, including model and non-model organisms. These transformer models yield transferable, context-specific representations of nucleotide sequences, which allow for accurate molecular phenotype prediction even in low-data settings. We show that the representations alone match or outperform specialized methods on 11 of 18 prediction tasks, and up to 15 after fine-tuning. Despite no supervision, the transformer models learnt to focus attention on key genomic elements, including those that regulate gene expression, such as enhancers. Lastly, we demonstrate that utilizing model representations alone can improve the prioritization of functional genetic variants. The training and application of foundational models in genomics explored in this study provide a widely applicable stepping stone to bridge the gap of accurate molecular phenotype prediction from DNA sequence alone.

Species-aware DNA language models capture regulatory elements and their evolution

“…Parallel to our work, another study was released which also trains on genomes of multiple species and focuses on human for downstream tasks (Dalla-Torre et al .,2023). The proposed model is species-agnostic but more than one thousand times larger than ours.…”

Fishing for a reelGene: evaluating gene models with evolution and machine learning

“…This is where the integration of protein language model ProtTrans has significantly enhanced the utility and effectiveness of the reelGene framework and opened up new avenues for protein functionality research. DNA language models are rapidly advancing, but currently, they are mostly pre-trained on human 54 and vertebrate genomes 55 . Models pre-trained on plant genomes may improve our mRNA and junction boundary models.…”

A Foundational Large Language Model for Edible Plant Genomes

“…Although effective these architectures have been outperformed by transformer architectures as evidenced by their adoption in both the computer vision and natural language processing fields [15,16,17]. To address these limitations and exploit the transformer architecture, large language models (LLMs) have gained considerable popularity in the field of biology [18,19,20,21,22,23,24,25,26,27], offering the ability to be trained on unlabeled data and gener-ate general-purpose representations capable of solving specific tasks. Furthermore, LLMs overcome a current limitation of other deep learningbased models, as they are not reliant on single reference genomes, which often provide an incomplete and biased genomic diversity depiction from a limited number of individuals.…”

“…Furthermore, LLMs overcome a current limitation of other deep learning-based models, as they are not reliant on single reference genomes, which often provide an incomplete and biased genomic diversity depiction from a limited number of individuals. LLMs can leverage multiple reference genomes, including those from genetically distant species, thereby increasing overall diversity, which has been shown to significantly enhance prediction performance [24]. This diversity is particularly relevant in plant species due to the structural complexities of their genomes, which hinder accurate mapping of polymorphisms across whole-genome alignments.…”