Strategies for effectively modelling promoter-driven gene expression using transfer learning

Abstract

Advances in gene delivery technologies are enabling rapid progress in molecular medicine, but require precise expression of genetic cargo in desired cell types, which is predominantly achieved via a regulatory DNA sequence called a promoter; however, only a handful of cell type-specific promoters are known. Efficiently designing compact promoter sequences with a high density of regulatory information by leveraging machine learning models would therefore be broadly impactful for fundamental research and direct therapeutic applications. However, models of expression from such compact promoter sequences are lacking, despite the recent success of deep learning in modelling expression from endogenous regulatory sequences. Despite the lack of large datasets measuring promoter-driven expression in many cell types, data from a few well-studied cell types or from endogenous gene expression may provide relevant information for transfer learning, which has not yet been explored in this setting. Here, we evaluate a variety of pretraining tasks and transfer strategies for modelling cell type-specific expression from compact promoters and demonstrate the effectiveness of pretraining on existing promoter-driven expression datasets from other cell types. Our approach is broadly applicable for modelling promoter-driven expression in any data-limited cell type of interest, and will enable the use of model-based optimization techniques for promoter design for gene delivery applications. Our code and data are available at https://github.com/anikethjr/promoter_models.

Designing DNA With Tunable Regulatory Activity Using Discrete Diffusion

“…A separate model was trained for each cell line. Notably, the performance of ResidualBind for K562 (Pearson’s r = 0.831), HepG2 (Pearson’s r = 0.839), and SK-N-SH (Pearson’s r = 0.824) was higher than what was achieved by the models explored in [100]. Nevertheless, nuances in data processing and training make direct comparisons not straightforward (even with the same data splits).…”

Designing Cell-Type-Specific Promoter Sequences Using Conservative Model-Based Optimization

“…(Zhou and Troyanskaya, 2015; Agarwal and Shendure, 2020; Avsec et al, 2021)). Reddy et al (2024) find that a model consisting of convolutional layers followed by transformer layers (MTLucifer) outperforms other architectures when used to model PE without any transfer learning.…”

“…This allows us to use one design model to design cell-type-specific promoters for all c ∈ C . In this section, we provide guidelines to build these models based on the findings of Reddy et al (2024), which studies model architectures and transfer learning strategies for modelling PE.…”

“…In such data-constrained settings, transfer learning can be very beneficial. Indeed, Reddy et al (2024) showed that pretraining a sequence-based model on large related genomic datasets before fine-tuning it on a smaller PE dataset from target cells leads to significantly better modelling of the smaller dataset compared to training exclusively on the smaller dataset. The three best approaches that they identified are:…”

“…Here we present a systematic framework and case study for applying offline MBO to the real world problem of promoter design, backed by empirical evidence from wetlab experiments evaluating differential expression in three closely related blood-based cell types. To improve data-efficiency through transfer learning, we follow previous work (Reddy et al, 2024) and pretrain sequence-based deep learning (DL) models of PE using large existing MPRA datasets that measure PE in different contexts, and fine-tune them using a small PE dataset collected from the target cell types. To address adversarial designs, we extend the conservative objective models (COMs) framework (Trabucco et al, 2021) to our setting: while fine-tuning models, we use a loss term that reduces the predicted DE of adversarial designs.…”