Generalized enhanced suffix array construction in external memory

Abstract

BackgroundSuffix arrays, augmented by additional data structures, allow solving efficiently many string processing problems. The external memory construction of the generalized suffix array for a string collection is a fundamental task when the size of the input collection or the data structure exceeds the available internal memory.ResultsIn this article we present and analyze [introduced in CPM (External memory generalized suffix and arrays construction. In: Proceedings of CPM. pp 201–10, 2013)], the first external memory algorithm to construct generalized suffix arrays augmented with the longest common prefix array for a string collection. Our algorithm relies on a combination of buffers, induced sorting and a heap to avoid direct string comparisons. We performed experiments that covered different aspects of our algorithm, including running time, efficiency, external memory access, internal phases and the influence of different optimization strategies. On real datasets of size up to 24 GB and using 2 GB of internal memory, showed a competitive performance when compared to and , which are efficient algorithms for a single string according to the related literature. We also show the effect of disk caching managed by the operating system on our algorithm.ConclusionsThe proposed algorithm was validated through performance tests using real datasets from different domains, in various combinations, and showed a competitive performance. Our algorithm can also construct the generalized Burrows-Wheeler transform of a string collection with no additional cost except by the output time.

Metagenomic analysis through the extended Burrows-Wheeler transform

“…The Burrows-Wheeler Transform (BWT) [ 36 ] is a well-known widely used reversible string transformation that can be extended to a collection of strings. Such an extension, known as eBWT or multi-string BWT, is a reversible transformation whose output string (denoted by ) is a permutation of the symbols of all strings in [ 21 ] (see also [ 30 , 31 , 34 , 47 , 48 ]). The length of is denoted by , and , with 1 ≤ i ≤ N , if x circularly precedes the i -th suffix S j [ k , n j +1] (for some 1≤ j ≤ m and 1≤ k ≤ n j + 1), according to the lexicographic sorting of the suffixes of all strings in .…”

“…A [36] is a well-known widely used reversible string transformation that can be extended to a collection of strings. Such an extension, known as eBWT or multi-string BWT, is a reversible transformation whose output string (denoted by ebwt(S)) is a permutation of the symbols of all strings in S [21] (see also [30,31,34,47,48]). The length of ebwt(S) is denoted by N = m i=1 n i + m, and The set S will be omitted if it is clear from the context.…”

External memory BWT and LCP computation for sequence collections with applications

“…The tool BCR+LCP cannot handle input sequences of different lengths, so for the pacbio dataset we used the tool extLCP [10] by the same authors. As a reference we also tested the external memory tool eGSA [23] that computes the Suffix and LCP arrays for a collection of sequences. However, we tested eGSA only using 32GB of RAM since the authors in [23] showed its running time degrades about 25 times when the RAM is restricted to the input size.…”

“…space economical, algorithms was recognized since the very beginning of the field [8,26,27]. When even lightweight algorithms do not fit in RAM, one has to resort to external memory construction algorithms (see [13,18,19,23] and references therein).…”

The Colored Longest Common Prefix Array Computed via Sequential Scans

“…The longest common prefix (LCP) array of the collection S [30,18,24] is the array lcp(S) of length N + 1, such that lcp(S)[i], with 2 ≤ i ≤ N , is the length of the longest common prefix between the suffixes associated to the positions i and i − 1 in ebwt(S) and lcp(S)[1] = lcp(S)[N + 1] = −1 set by default. We denote by LCP(i, j) the length of the LCP between the suffixes associated with positions i and j in ebwt(S), i.e.…”