A Foundational Large Language Model for Edible Plant Genomes

Abstract

In recent years, significant progress has been made in the field of plant genomics, demonstrated by the increased use of high-throughput methodologies that allow for the characterization of multiple genome-wide molecular phenotypes. These results have provided valuable insights into plant traits and their underlying genetic mechanisms, especially in well-researched model plant species. Nonetheless, although acquiring and characterizing these molecular phenotypes can offer valuable insights into plant traits, effectively leveraging them to make accurate predictions represents a critical step in crop genomic improvement. We present AgroNT, a novel foundational large language model trained on reference genomes from 48 plant species with a predominant focus on crop species. We show that AgroNT can obtain state-of-the-art predictions for many genomic elements, including polyadenylation sites, splice sites, open chromatin and enhancer regions. Furthermore, AgroNT can be used to predict the strength of promoter sequences and tissue-specific gene expression levels or prioritize functional variants. Using the cassava genome as an example of an understudied species, we perform a large-scale in silico saturation mutagenesis analysis to assess the impact of >10 million mutations on gene expression levels and enhancer elements in the cassava genome, and provide the results as a valuable resource for regulatory causal variant characterization. Furthermore, owing to the lack of comprehensive benchmarks in the context of deep learning-based methods in plant genomic research, we propose the use of the multiple datasets encompassing seven distinct genomic prediction tasks, which have been compiled here, as the Plants Genomic Benchmark (PGB). The pre-trained AgroNT model is now publicly available on Hugging Face at https://huggingface.co/InstaDeepAI/agro-nt for future research purposes.

Current genomic deep learning architectures generalize across grass species but not alleles

“…Based on theory and prior interpretation work on expression models (Mendoza-Revilla et al, 2023), we hypothesized our expression models would also pay most attention to the region surrounding the transcription start site. Looking at the average saliency map for DanQ across all B73 genes on the maximum expression task we see that DanQ indeed focuses on the core promoter region and the 5′ UTR (Figure 4, right).…”

“…FNetCompression's performance is particularly remarkable because it has several orders of magnitude fewer parameters than DanQ (57k versus 1.6m, respectively). Large foundation models such as AgroNT (Mendoza-Revilla et al, 2023) show promising results within the training species, but FNetCompression suggests smaller, more efficient models, perhaps also utilizing a fast Fourier transform, are worth further exploration. Since the Pearson correlations we observed are still far from perfect, it is worthwhile to note that we do not expect cis sequence-based models to ever reach perfect correlation, as cis factors explain only a third of the genetic variation in expression in maize (Giri et al, 2021).…”

Cross-species modeling of plant genomes at single nucleotide resolution using a pre-trained DNA language model

“…Therefore, we generated three extremely imbalanced cross-species testing datasets for rice, sorghum, and maize using BUSCOsupported genes as true sites (Figure 2B and Supplemental Table 3). We benchmarked the performance of PlantCaduceus against three DNA LMs: GPN 16 , AgroNT 18 , and Nucleotide…”

“…To comprehensively evaluate our foundation model's performance, four foundation models including GPN 16 , custom GPN, AgroNT 18 and NT-v2 19 were used as baselines for various tasks.…”

“…By down-sampling noncoding regions and down-weighting repetitive sequences, we generated an unbiased genomic dataset for pre-training. In contrast, other publicly available DNA LMs, such as AgroNT 18 and Nucleotide Transformer 19 , use the entire genomes for pre-training, potentially introducing biases toward certain genomes and repetitive sequences. Unlike the unidirectional HyenaDNA 20 or Evo 21 , or the convolutional neural network-based GPN 16 , PlantCaduceus offers bi-directional context, providing a more comprehensive understanding of DNA interactions.…”