Sparse Binary Relation Representations for Genome Graph Annotation

Abstract

High-throughput DNA sequencing data is accumulating in public repositories, and efficient approaches for storing and indexing such data are in high demand. In recent research, several graph data structures have been proposed to represent large sets of sequencing data and to allow for efficient querying of sequences. In particular, the concept of labeled de Bruijn graphs has been explored by several groups. While there has been good progress towards representing the sequence graph in small space, methods for storing a set of labels on top of such graphs are still not sufficiently explored. It is also currently not clear how characteristics of the input data, such as the sparsity and correlations of labels, can help to inform the choice of method to compress the graph labeling. In this work, we present a new compression approach, Multi-BRWT, which is adaptive to different kinds of input data. We show an up to 29% improvement in compression performance over the basic BRWT method, and up to a 68% improvement over the current state-of-the-art for de Bruijn graph label compression. To put our results into perspective, we present a systematic analysis of five different state-of-the-art annotation compression schemes, evaluate key metrics on both artificial and real-world data and discuss how different data characteristics influence the compression performance. We show that the improvements of our new method can be robustly reproduced for different representative real-world datasets.

Building large updatable colored de Bruijn graphs via merging

“…Lastly, there are a couple of methods that construct the color matrix in a manner that is both compressed and dynamic, which include the method of Mustafa et al (2019) and Multi-BRWT (Karasikov et al , 2019). These methods use a simple representation of a de Bruijn graph where each edge (the k -mer) is stored in a hash table.…”

“…In Section 5, we compared the performance of VariMerge against Vari (Muggli et al , 2017)/Rainbowfish (Almodaresi et al , 2017), Bloom Filter Trie (Holley et al , 2016), Multi-BRWT (Karasikov et al , 2019) and Mantis (Pandey et al , 2018)/the method of Almodaresi et al (2019). Among all mentioned methods in this section, we chose these tools based on the following criteria: the method should be both graph-based and non-probabilistic (exact) for a fair comparison.…”

“…A bottleneck in applying Cortex (Iqbal et al , 2012) to large datasets lies in the amount of memory and CPU time required to build and store the colored de Bruijn graph. Thus, Vari (Muggli et al , 2017), Rainbowfish (Almodaresi et al , 2017), Mantis (Almodaresi et al , 2019), the method of Karasikov et al (2019), as well as others, sought to overcome this limitation by building and/or storing the colored de Bruijn graphs in a space- and time-efficient manner. Vari was one of the first methods to build the colored de Bruijn graph in a memory efficient manner.…”

“…For example, Vari (Muggli et al , 2017), Rainbowfish (Almodaresi et al , 2017), Mantis (Pandey et al , 2018) and the method of Almodaresi et al (2019) are efficient with respect to memory and time but cannot be updated without reconstructing the entire colored de Bruijn graph. Bloom Filter Trie (Holley et al , 2016) and the method of Karasikov et al (2019) allow for the addition of new data but cannot scale to large datasets, as we show in this article. Crawford et al (2018) build a de Bruijn graph that can be updated but restrict interest to the traditional (non-colored) de Bruijn graph.…”

“…This construction required 254 GB of RAM, 2.34 TB of external memory, and ∼140 h of CPU time. To contextualize these results, we compare the construction of VariMerge with those of state-of-the-art methods, including Vari (Muggli et al , 2017)/Rainbowfish (Almodaresi et al , 2017), Bloom Filter Trie (Holley et al , 2016), Mantis (Pandey et al , 2018)/the method of Almodaresi et al (2019) and the method of Karasikov et al (2019), on datasets consisting of 4000, 8000 and 16 000 strains. Further, we demonstrate the ability of the graph to be updated efficiently through VariMerge by showing a Salmonella strain can be added to the graph in an order of magnitude faster than its initial construction.…”