Linear-Size CDAWG: New Repetition-Aware Indexing and Grammar Compression

Takagi, Toshihisa; Goto, Keisuke; Fujishige, Yuta; Inenaga, Shunsuke; Arimura, Hiroki

doi:10.1007/978-3-319-67428-5_26

Cited by 18 publications

(29 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By using O(r log log w (σ + n/r)) space, we obtain optimal locate time in the general setting, O(m + occ), as well as optimal counting time, O(m). This had been obtained before only with space bounds O(e) [7] or O(e) [111]. 4.…”

mentioning

confidence: 70%

See 1 more Smart Citation

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

123

142

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transforms (BWTs). One of the earliest indexes for repetitive collections, the Run-Length FM-index, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitivenessthe number of phrases in the Lempel-Ziv parse, the size of the smallest grammar generating (only) the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FM-index so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time, O(m + occ), within O(r log log w (σ + n/r)) space, for a text of length n over an alphabet of size σ on a RAM machine with words of w = Ω(log n) bits. Within that space, our index can also count in optimal time, O(m). Multiplying the space by O(w/ log σ), we support count and locate in O( m log(σ)/w ) and O( m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and extracts any text substring of length in almost-optimal time O(log(n/r) + log(σ)/w). Within that space, we similarly provide direct access to suffix array, inverse suffix array, and longest common prefix array cells, and extend these capabilities to full suffix tree functionality, typically in O(log(n/r)) time per operation. Our experiments show that our O(r)-space index outperforms the space-competitive alternatives by 1-2 orders of magnitude.

show abstract

mentioning

confidence: 70%

“…4. Self-indexes with efficient extraction require Ω(z log(n/z)) space [105,21,43,10,15], Ω(g) space [17,14], or Ω(e) space [111,7]. 5.…”

Section: Indexmentioning

confidence: 99%

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

123

142

View full text Add to dashboard Cite

show abstract

“…20] O(r + n/s) O(( + s)( log σ log log r + (log log n) 2 )) This paper (Thm. 2) O(r log(n/r)) O(log(n/r) + log(σ)/w) Takagi et al [94,Thm. 9] O(e) O(log n + ) Belazzougui O(r log(n/r)) O(log(n/r) + log(σ)/w) Access SA, ISA, LCP (Thm.…”

Section: Index Spacementioning

confidence: 99%

Optimal-Time Text Indexing in BWT-runs Bounded Space

Gagie¹,

Navarro²,

Prezza³

2018

Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms

126

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transform (BWT). One of the earliest indexes for repetitive collections, the Run-Length FMindex, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitiveness -the number of phrases in the LempelZiv parse, the size of the smallest grammar generating the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FMindex so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time O(m + occ) within O(r log(n/r)) space, on a RAM machine with words of w = Ω(log n) bits. Raising the space to O(rw log σ (n/r)), we support locate in O(m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and efficiently extracts any text substring, with an O(log(n/r)) additive time penalty over the optimum. Preliminary experiments show that our new structure outperforms the alternatives by orders of magnitude in the space/time tradeoff map.

show abstract

“…to edges in the suffix tree of T . This parallels the grammar implicit in [6] and explicit in [21], whose nonterminals correspond to unary paths in the suffix trie of T , i.e. to edges in the suffix tree of T .…”

Section: Cdawgmentioning

confidence: 70%

“…We achieve this by dropping the run-length-encoded representation of the Burrows-Wheeler transform of T , used in [2], and by exploiting the fact that the reversed CDAWG induces a context-free grammar that produces T and only T , as described in [1]. A related grammar, already implicit in [6], has been concurrently exploited in [21] to achieve similar bounds to ours. Note that in some strings, for example in the family T i for i ≥ 0, where T 0 = 0 and T i = T i−1 iT i−1 , the length of the string grows exponentially in the size of the CDAWG, thus shaving an O(log log n) term is identical to shaving an O(log e T ) term.…”

Section: Introductionmentioning

confidence: 99%

Fast Label Extraction in the CDAWG

Belazzougui

Cunial

2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The compact directed acyclic word graph (CDAWG) of a string T of length n takes space proportional just to the number e of right extensions of the maximal repeats of T , and it is thus an appealing index for highly repetitive datasets, like collections of genomes from similar species, in which e grows significantly more slowly than n. We reduce from O(m log log n) to O(m) the time needed to count the number of occurrences of a pattern of length m, using an existing data structure that takes an amount of space proportional to the size of the CDAWG. This implies a reduction from O(m log log n + occ) to O(m + occ) in the time needed to locate all the occ occurrences of the pattern. We also reduce from O(k log log n) to O(k) the time needed to read the k characters of the label of an edge of the suffix tree of T , and we reduce from O(m log log n) to O(m) the time needed to compute the matching statistics between a query of length m and T , using an existing representation of the suffix tree based on the CDAWG. All such improvements derive from extracting the label of a vertex or of an arc of the CDAWG using a straight-line program induced by the reversed CDAWG.

show abstract

Linear-Size CDAWG: New Repetition-Aware Indexing and Grammar Compression

Cited by 18 publications

References 20 publications

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Optimal-Time Text Indexing in BWT-runs Bounded Space

Fast Label Extraction in the CDAWG

Contact Info

Product

Resources

About