Succinct Dynamic de Bruijn Graphs

Abstract

Motivation:The de Bruijn graph is one of the fundamental data structures for analysis of high throughput sequencing data. In order to be applicable to population-scale studies, it is essential to build and store the graph in a space-and time-efficient manner. In addition, due to the ever-changing nature of population studies, it has become essential to update the graph after construction e.g. add and remove nodes and edges. Although there has been substantial effort on making the construction and storage of the graph efficient, there is a limited amount of work in building the graph in an efficient and mutable manner. Hence, most space efficient data structures require complete reconstruction of the graph in order to add or remove edges or nodes. Results: In this paper we present DynamicBOSS, a succinct representation of the de Bruijn graph that allows for an unlimited number of additions and deletions of nodes and edges. We compare our method with other competing methods and demonstrate that DynamicBOSS is the only method that supports both addition and deletion and is applicable to very large samples (e.g. greater than 15 billion k-mers). Competing dynamic methods e.g., FDBG (Crawford et al., 2018) cannot be constructed on large scale datasets, or cannot support both addition and deletion e.g., BiFrost

Indexing All Life’s Known Biological Sequences

“…Second, a graph can be directly updated if it is represented using dynamic data structures (e.g., SuccinctDBG in the dynamic form), which support dynamic updates. This approach allows making instant changes, however, does not enable large updates because of the limited performance of dynamic data structures [2]. Finally, for large updates, the existing index can be completely reconstructed.…”

Buffering Updates Enables Efficient Dynamic de Bruijn Graphs

“…Here, we give some background on this modification. Alipanahi et al (2020a) observed that FDBG fails for datasets approaching 2 32 distinct nodemers. This is due to hash collisions.…”

“…Munro et al also explained how their ideas could be used to obtain dynamic compact representations of graphs, and Coimbra et al (2020) recently applied those ideas to obtain an implementation of dynamic k 2 -trees that is competitive with previous implementations, which were essentially just static k 2 -trees but with dynamic instead of static bitvectors. Since Dynamic-BOSS (Alipanahi et al, 2020a) is essentially the data structure of Bowe et al (2012) but with dynamic instead of static bit vectors, we are naturally curious how competitive a dynamic de Bruijn graph based on Munro et al's ideas will be.…”

“…Even allowing for the addition of data into a compressed version of the de Bruijn graph requires some algorithmic cleverness; most notably, VariMerge (Muggli et al, 2019) and Bifrost (Holley and Melsted, 2020) allow only addition of new data. Hence, there are relatively few compressed data structures for storing the de Bruijn graph that allow for the addition and deletion of data; these include FDBG (Crawford et al, 2018) and Dy-namicBOSS (Alipanahi et al, 2020a). In Table 1, we give an overview of the basic de Bruijn graph data structures and the operations that they support.…”

“…In particular, with the exception of a few methods (Muggli et al, 2019; Holley and Melsted, 2020; Crawford et al, 2018), compressed and compact de Bruijn graphs do not allow for the graph to be efficiently updated, allowing for data to be added or deleted. The most recent compressed dynamic de Bruijn graph, (Alipanahi et al, 2020a), relies on dynamic bit vectors, which are slow in theory and practice. To address this shortcoming, we present BufBOSS which is a compressed dynamic de Bruijn graph that removes the necessity of dynamic bit vectors by buffering data that should be added or removed from the graph.…”