On the application of BERT models for nanopore methylation detection

Abstract

DNA methylation is a common epigenetic modification, which is widely associated with various biological processes, such as gene expression, aging, and disease. Nanopore sequencing provides a promising methylation detection approach through monitoring abnormal signal shifts for detecting modified bases in target motif regions. Recently, model-based approaches, especially those with deep learning models, have achieved significant performance improvements on nanopore methylation detection. In this work, we explore using bidirectional encoder representations from transformers (BERT) for doing the task, which can provide non-recurrent neural structures for fast parallel computation. We find original BERT architecture does not work as well as the bidirectional recurrent neural network (biRNN) on the nanopore methylation prediction task. Through further analysis, we observe recurrent patterns of positional-signal-shift in the context window surrounding target 5-methylcytosine (5mC) and N6-methyladenine (6mA) motifs. We propose a refined BERT with relative position representation and center hidden units concatenation, which takes account of task-specific characters into modeling. We perform systematic evaluations in-sample and cross-sample. The experiment results show that the refined BERT model can achieve competitive or even better results than the state-of-the-art biRNN model, while the model inference speed is about 6x faster. Besides, on the cross-sample evaluation of datasets from the different research groups, BERT models demonstrate a good generalization performance.

DNA methylation-calling tools for Oxford Nanopore sequencing: a survey and human epigenome-wide evaluation

“…Several different versions of nanopore chemistry have been developed by ONT to improve the accuracy of single-molecule sequencing (Fig. 1A [9,[12][13][14][15][16][17][18][19][20][21][22][23]). Both the first pore version, termed R6 ("R" for Reader), and the subsequent R7 pore series yielded high error rates and only mediocre accuracy [11].…”

“…Specifically, the electric current patterns, also known as "squiggles," resulting from the passage of modified bases through the pores differs from the patterns produced by the passage of unmodified bases [26,30]. The difference can be determined after nanopore read basecalling and alignment by (1) statistical tests comparing the electric current pattern to an in silico reference or the pattern from a nonmodified control sample [20,31]; (2) pre-trained supervised learning models, e.g., neural network [23,[32][33][34][35][36][37], machine learning model [38], and Hidden Markov Models (HMM) [9,39]. However, DNA-methylation detection using nanopore sequencing presents a methodological challenge, i.e., the capacity to detect modifications in different CpGs that are in close proximity to one another on a DNA fragment (i.e., nonsingleton), as it is assumed that all CpGs within a 10-bp region share the same methylation status.…”

“…However, DNA-methylation detection using nanopore sequencing presents a methodological challenge, i.e., the capacity to detect modifications in different CpGs that are in close proximity to one another on a DNA fragment (i.e., nonsingleton), as it is assumed that all CpGs within a 10-bp region share the same methylation status. Twelve methylation-calling tools have been developed for various DNA modifications (e.g., 4mC, 5mC, 5hmC, and 6 mA) and for different nanopore pore versions (e.g., R7, R9, and R10) (Table 1 [9,20,23,[31][32][33][34][35][36][37][38][39]), but DNA-methylation detection for non-singletons containing both methylated and unmethylated CpGs remains difficult [9,35]. Moreover, DNA methylation levels are not linearly distributed across the genome, and CpG density is dependent on genomic context [40][41][42].…”

“…Methylation-calling tool are colored by the methylation calling methods (Green: statistical tests, Purple: HMM, Orange: neural network, white: machine learning models). Relevant publication dates are from multiple source [9,[12][13][14][15][16][17][18][19][20][21][22][23]. B Workflow for 5-methylcytosine (5mC) detection for nanopore sequencing.…”

MuLan-Methyl - Multiple Transformer-based Language Models for Accurate DNA Methylation Prediction

“…Here, we phrase DNA methylation-site detection as a Natural Language Processing (NLP) problem and propose a novel framework to address it. Previous studies for identifying methylation sites usually use BERT, a classic NLP approach, or, in the context of DNA sequences, the variant DNABERT (36), either as a model that accepts embeddings from Word2vec, or as an encoder that generates embeddings for input to a deep neural network (23, 25, 32, 33, 37).…”