Representation learning of genomic sequence motifs with convolutional neural networks

Abstract

Although convolutional neural networks (CNNs) have been applied to a variety of computational genomics problems, there remains a large gap in our understanding of how they build representations of regulatory genomic sequences. Here we perform systematic experiments on synthetic sequences to reveal how CNN architecture, specifically convolutional filter size and max-pooling, influences the extent that sequence motif representations are learned by first layer filters. We find that CNNs designed to foster hierarchical representation learning of sequence motifs-assembling partial features into whole features in deeper layers-tend to learn distributed representations, i.e. partial motifs. On the other hand, CNNs that are designed to limit the ability to hierarchically build sequence motif representations in deeper layers tend to learn more interpretable localist representations, i.e. whole motifs. We then validate that this representation learning principle established from synthetic sequences generalizes to in vivo sequences.

Interpreting Deep Neural Networks Beyond Attribution Methods: Quantifying Global Importance of Genomic Features

“…To generate data-driven hypotheses, first-order and secondorder attribution methods can be employed to identify important local features. Because attribution maps can be noisy, it may be beneficial to employ CNNs that are designed to learn more interpretable representations in first layer filters (Koo & Eddy, 2019). It turns out that CNNs designed to learn interpretable filters also yield more reliable representations with attribution methods .…”

“…Recent advances have made it possible to intentionally design CNNs to learn more human-interpretable patterns in convolutional filters. This includes design principles based on spatial information flow through the network and employing highly divergent activation functions such as the exponential function (Koo & Eddy, 2019;Koo & Ploenzke, 2019). In parallel, advances have been developed to make direct weight visualization more interpretable (Ploenzke & Irizarry, 2018).…”

Deep Learning for RNA Synthetic Biology

“…These results suggest that while the use of rational features may facilitate the abstraction of potentially relevant information of toehold switch function, the one-hot sequence-only MLP model can recover such information without a priori hypothesisdriven assumptions built into the model if given sufficient training data. 40 In order to evaluate the degree of biological generalization in our sequence-only MLP model, we performed two additional rounds of validation. First, we iteratively withheld each of the 23 tiled viral genomes in the dataset during training and predicted their function as test sets, resulting in a 0.82-0.98 AUROC range (average 0.87, Fig.…”

“…By contrast, mechanistic hypothesis-driven models can more directly inform which aspects of a biological theory best explain the observations. Various methods have been established to address this limitation, including alternative network 30 architectures (39), and the use of saliency maps (40,41), which reveal the regions of an input that deep learning models weigh most heavily and therefore pay the most attention to when making predictions. While saliency maps have been previously used to visualize model attention in one-hot representations of sequence data (10,17,18,20,40), such implementations focus only on the primary sequence and have not been developed to identify secondary structure 35 interactions, which are especially relevant in the operation of RNA synthetic biology elements.…”

“…In the few cases where secondary structure has been investigated, input representations have been constrained to predetermined structures based on the predictions of thermodynamic models (37, 38) whose abstractions we have found cause significant information loss. 40 We sought to visualize RNA secondary structures learned by our neural networks in a manner unconstrained by thermodynamic modeling. To achieve this, we trained a CNN on a twodimensional nucleotide complementarity map representation (Fig.…”