Bloom Filter Trie: an alignment-free and reference-free data structure for pan-genome storage

Abstract

BackgroundHigh throughput sequencing technologies have become fast and cheap in the past years. As a result, large-scale projects started to sequence tens to several thousands of genomes per species, producing a high number of sequences sampled from each genome. Such a highly redundant collection of very similar sequences is called a pan-genome. It can be transformed into a set of sequences “colored” by the genomes to which they belong. A colored de Bruijn graph (C-DBG) extracts from the sequences all colored k-mers, strings of length k, and stores them in vertices.ResultsIn this paper, we present an alignment-free, reference-free and incremental data structure for storing a pan-genome as a C-DBG: the bloom filter trie (BFT). The data structure allows to store and compress a set of colored k-mers, and also to efficiently traverse the graph. Bloom filter trie was used to index and query different pangenome datasets. Compared to another state-of-the-art data structure, BFT was up to two times faster to build while using about the same amount of main memory. For querying k-mers, BFT was about 52–66 times faster while using about 5.5–14.3 times less memory.ConclusionWe present a novel succinct data structure called the Bloom Filter Trie for indexing a pan-genome as a colored de Bruijn graph. The trie stores k-mers and their colors based on a new representation of vertices that compress and index shared substrings. Vertices use basic data structures for lightweight substrings storage as well as Bloom filters for efficient trie and graph traversals. Experimental results prove better performance compared to another state-of-the-art data structure.Availabilityhttps://www.github.com/GuillaumeHolley/BloomFilterTrie.

Simplitigs as an efficient and scalable representation of de Bruijn graphs

“…The resulting structure is usually referred to as a colored de Bruijn graph [19] and its representations have been widely studied ( [50][51][52][53][54][55][56][57][58][59][60][61] ). Even though we touched this setting in the section Multiple pan-genomes, exploiting the similarity between individual de Bruijn graphs for further compression in simplitig-based approaches is to be addressed in future work.…”

Representation ofk-mer sets using spectrum-preserving string sets

“…The second is BOSS, which, as mentioned previously, was shown [35] to have superior space usage. We did not compare against the Bloom filter trie [36], which is fast but uses an order of Table 2: Space usage of UST-Compress and others. We show the average number of bits per distinct k-mer in the dataset.…”

“…Membership data structures for k-mer sets were surveyed in a recent paper [9]. In addition to the unitig-based approaches already mentioned, other exact representations include succinct de Bruijn graphs (referred to as BOSS [36]) and their variations [37,38], dynamic de Bruijn graphs [39,40], and Bloom filter tries [41]. Some data structures are non-static, i.e.…”

“…The second is BOSS, which, as mentioned previously, was shown [40] to have superior space usage. We did not compare against the Bloom filter trie [41], which is fast but uses an order of magnitude more memory than BOSS [40]. Other data structures, such as Pufferfish [15], blight [16], and Bifrost [17], implement more sophisticated operations and hence use significantly more memory than BOSS.…”

Succinct Dynamic de Bruijn Graphs

“…Hence, this allows the matrix to be compressed and stored independently of the graph. Also several other methods have been developed to further compress, store, and manipulate the color matrix, including Rainbowfish (Almodaresi et al, 2017), Mantis (Pandey et al, 2018), Bloom Filter Trie (BFT) (Holley et al, 2016), and Bifrost (Holley and Melsted, 2019).…”