MuLan-Methyl - Multiple Transformer-based Language Models for Accurate DNA Methylation Prediction

Abstract

Transformer-based language models are successfully used to address massive text-related tasks. DNA methylation is an important epigenetic mechanism and its analysis provides valuable insights into gene regulation and biomarker identification. Several deep learning-based methods have been proposed to identify DNA methylation and each seeks to strike a balance between computational effort and accuracy. Here, we introduce a MuLan-Methyl, a deep learning framework for predicting DNA methylation sites, which is based on multiple (five) popular transformer-based language models. The framework identifies methylation sites for three different types of DNA methylation, namely N6-adenine (6mA), N4-cytosine (4mC), and 5-hydroxymethylcytosine (5hmC). Each of the five employed language models is adapted to the task using the "pre-train and fine-tune" paradigm. Pre-training is performed on a custom corpus consisting of DNA fragments and taxonomy lineages using self-supervised learning. Fine-tuning then aims at predicting the DNA-methylation status of each type. The five models are used to collectively predict the DNA methylation status. We report excellent performance of MuLan-Methyl on a benchmark dataset. Moreover, we show that the model captures characteristic differences between different species that are relevant for methylation. This work demonstrates that language models can be successfully adapted to this domain of application and that joint utilization of different language models improves model performance.

iLEC-DNA: Identifying Long Extra-chromosomal Circular DNA by Fusing Sequence-derived Features of Physicochemical Properties and Nucleotide Distribution Patterns

“…Accuracy 23 refers to the proportion of correct predictions with respect to the total predictions. Specificity or true negative rate (TNR) 20 is the model’s ability to correctly predict the negative class samples.…”

“…It is determined by dividing the number of correct negative predictions by the total number of true negatives. Sensitivity (or recall) 23 measures the ability of the model to predict positive class samples by taking the ratio of correct positive predictions to the predictions on positive samples. MCC 78 calculates the correlation between the model predictions and the true class, by taking into consideration true positives, true negatives, false positives, and false negatives.…”

“…The limitations of wet lab based methods and exceptional performance of AI based applications in natural language processing (NLP) tasks, have prompted a marathon of developing AI methods for DNA sequence analysis. Several AI models have been developed for various DNA analysis tasks such as enhancer identification 20,21 , DNA modification prediction 22,23 , promoter prediction 24 , DNA cyclizability prediction 25 , nucleosome position detection 26 and so on. On the other hand, the identification of eccDNA is still being performed through wet lab-based methods due to the deficiency of AI applications for this particular task.…”

iLEC-DNA: Identifying Long Extra-chromosomal Circular DNA by Fusing Sequence-derived Features of Physicochemical Properties and Nucleotide Distribution Patterns

“…It is determined by dividing the number of correct negative predictions by the total number of true negatives. Sensitivity (or recall) 23 In the mathematical expression above, T + and T − denote the true predictions related to positive and negative classes, whereas F + and F − are the incorrect predictions related to the positive and negative classes respectively.…”

Strategies for effectively modelling promoter-driven gene expression using transfer learning

“…For example, self-supervised tasks such as masked language modelling (MLM) have recently been used to pretrain genomic sequence embeddings that are then fine-tuned for downstream tasks (e.g. Ji et al (2021); Mo et al (2021); Benegas et al (2022); Zeng et al (2023)). Pretraining using task-relevant data can improve the performance of fine-tuned models (Gururangan et al, 2020), while pretraining using irrelevant data can hurt performance (Liu et al, 2022).…”