Lossless Indexing with Counting de Bruijn Graphs

Abstract

High-throughput sequencing data is rapidly accumulating in public repositories. Making this resource accessible for interactive analysis at scale requires efficient approaches for its storage and indexing. There have recently been remarkable advances in solving the experiment discovery problem and building compressed representations of annotated de Bruijn graphs where k-mer sets can be efficiently indexed and interactively queried. However, approaches for representing and retrieving other quantitative attributes such as gene expression or genome positions in a general manner have yet to be developed.In this work, we propose the concept of Counting de Bruijn graphs generalizing the notion of annotated (or colored) de Bruijn graphs. Counting de Bruijn graphs supplement each node-label relation with one or many attributes (e.g., a k-mer count or its positions in genome). To represent them, we first observe that many schemes for the representation of compressed binary matrices already support the rank operation on the columns or rows, which can be used to define an inherent indexing of any additional quantitative attributes. Based on this property, we generalize these schemes and introduce a new approach for representing non-binary sparse matrices in compressed data structures. Finally, we notice that relation attributes are often easily predictable from a node's local neighborhood in the graph. Notable examples are genome positions shifting by 1 for neighboring nodes in the graph, or expression levels that are often shared across neighbors. We exploit this regularity of graph annotations and apply an invertible delta-like coding to achieve better compression.We show that Counting de Bruijn graphs index k-mer counts from 2,652 human RNA-Seq read sets in representations over 8-fold smaller and yet faster to query compared to state-of-the-art bioinformatics tools. Furthermore, Counting de Bruijn graphs with positional annotations losslessly represent entire reads in indexes on average 27% smaller than the input compressed with gzip -9 for human Illumina RNA-Seq and 57% smaller for PacBio HiFi sequencing of viral samples. A complete joint searchable index of all viral PacBio SMRT reads from NCBI's SRA (152,884 read sets, 875 Gbp) comprises only 178 GB. Finally, on the full RefSeq collection, they generate a lossless and fully queryable index that is 4.4-fold smaller compared to the MegaBLAST index. The techniques proposed in this work naturally complement existing methods and tools employing de Bruijn graphs and significantly broaden their applicability: from indexing k-mer counts and genome positions to implementing novel sequence alignment algorithms on top of highly compressed and fully searchable graph-based sequence indexes.

On Weighted K-Mer Dictionaries

“…Lastly in this section, we report that other works [17, 14] considered the multi-document version of the problem studied here, that is, how to retrieve a vector of weights for a query k -mer, where each component of the vector represents the weight of the k -mer in a distinct document. Also such count vectors are usually very “regular” (or can be made so by introducing some approximation) [17] and present runs of equal symbols that can be compressed effectively with run-length encoding (RLE).…”

fimpera: drastic improvement of Approximate Membership Query data-structures with counts

“…At constant memory usage, adding the abundance information would yield an extremely high false-positive rate. As such, methods storing abundances mostly rely on compression by clustering abundance with neighbouring k-mers or across datasets, as Reindeer [10] or Counting de Bruijn graphs [6]. These methods do not rely on counting AMQ, but rather on exact data structures.…”

Label-guided seed-chain-extend alignment on annotated De Bruijn graphs

“…While a sequence graph by itself can be used to check for the presence or absence of a query sequence within a set, it cannot classify or profile the query without additional metadata, called graph annotations . Graph annotations are a key-value store associating each graph node with a number of annotations, where annotations can include the labels of the indexed samples [32,49,28,26], node abundances [33,44], genomic coordinates [33,1,20], geographic coordinates [32], etc. [59].…”

“…For jointly indexing unassembled read sets, annotated De Bruijn graphs typically scale better than variation graphs in representation size due to the collapse of shared k-mers between samples onto single graph nodes [32]. Before joint graph construction, many indexing tools [32,33,49,58,55,28,59,28] perform error correction ( cleaning ) on each sample to remove uncertain k-mers [62]. Thus, these joint graphs can represent the samples’ respective assembly graphs [32].…”

“…One of the inherent limitations of De Bruijn graphs is the intimate connection between the choice of k and the graph topology. Setting k too small leads to a graph topology where the majority of walks will spell sequences that are not present in the underlying genome and do not represent biologically-viable recombination [57] (called spurious walks [33]), while setting a large value will reduce the potential for shared substrings to collapse onto shared nodes, and hence, lead to lower coverage and more sequence content being removed during cleaning. Variable-order De Bruijn graphs [7] of maximum order k overcome this limitation by representing all observed k’-mers, where 1 < k′ ≤ k , including additional edges for transitioning between nodes of different k′ values as long as they share a common suffix.…”