Learning the histone codes with large genomic windows and three-dimensional chromatin interactions using transformer

Abstract

The quantitative characterization of the transcriptional control by histone modifications has been challenged by many computational studies, but most of them only focus on narrow and linear genomic regions around promoters, leaving a room for improvement. We present Chromoformer, a transformer-based, three-dimensional chromatin conformation-aware deep learning architecture that achieves the state-of-the-art performance in the quantitative deciphering of the histone codes in gene regulation. The core essence of Chromoformer architecture lies in the three variants of attention operation, each specialized to model individual hierarchy of transcriptional regulation involving from core promoters to distal elements in contact with promoters through three-dimensional chromatin interactions. In-depth interpretation of Chromoformer reveals that it adaptively utilizes the long-range dependencies between histone modifications associated with transcription initiation and elongation. We also show that the quantitative kinetics of transcription factories and Polycomb group bodies can be captured by Chromoformer. Together, our study highlights the great advantage of attention-based deep modeling of complex interactions in epigenomes.

Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction

“…Another potential limitation of SpliceBERT is that it could only process sequences no longer than 1024nt because the memory complexity of vanilla Transformer layers is quadratic to the sequence length. However, in many tasks in genomics, including splice site prediction, integrating large genomic context from thousands to millions of nucleotides is crucial to ensure the accuracy of predictions (9,(46)(47)(48). One straightforward solution is to adopt Transformers with linear memory complexity with respect to sequence length.…”

CREaTor: Zero-shotcis-regulatory pattern modeling with attention mechanisms

“…Since MYC is in both in-and cross-cell type test sets, we believe that CREaTor has learned general rules guiding cCRE-gene interactions in different cell types, rendering it an efficient tool for cCRE activity modeling in unseen cell types CREaTor captures chromatin domain boundaries in unseen cell types. Three-dimensional (3D) chromatin folding allows physical interactions between distal cCRE and genes and the information can also guide gene regulation modeling 13,28,42 . Without incorporating 3D chromatin folding information in our model, we were curious to see if CREaTor captured the topological structure of the genome, considering that CREaTor precisely recovers cCRE-gene interactions even of long ranges.…”

Investigating the performance of foundation models on human 3’UTR sequences

“…Trained in a self-supervised manner on vast amounts of genomic data, these models offer a comprehensive understanding of the sequence language and hold the potential to enable various downstream applications. Genomic foundation models have achieved encouraging results in promoter (Ji et al, 2021;Dalla-Torre et al, 2023;Zhou et al, 2023) and enhancer (Dalla-Torre et al, 2023) prediction, chromatin state analysis (Lee et al, 2022), transcription factor binding sites prediction (Ji et al, 2021;Dalla-Torre et al, 2023;Zhou et al, 2023), and functional variants prioritization (Ji et al, 2021;Dalla-Torre et al, 2023). The DNABERT model (Ji et al, 2021) was one of the first adaptations of large language models to the human genome.…”