LZ-End Parsing in Compressed Space

Kempa, Dominik; Kosolobov, Dmitry

doi:10.1109/dcc.2017.73

Cited by 17 publications

(18 citation statements)

References 15 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their search procedure, however, has an anomaly [34,Lem. 5]: in some cases it might return a child of v instead of v. We can fix this problem (and always return the correct v) by also storing, for every internal trie node v, the fingerprint of v.str and the exit character v.str[|v.par.str| + 1].…”

Section: Building the Multisets X And Ymentioning

confidence: 99%

Universal compressed text indexing

Navarro

Prezza

2019

Theoretical Computer Science

View full text Add to dashboard Cite

The rise of repetitive datasets has lately generated a lot of interest in compressed self-indexes based on dictionary compression, a rich and heterogeneous family of techniques that exploits text repetitions in different ways. For each such compression scheme, several different indexing solutions have been proposed in the last two decades. To date, the fastest indexes for repetitive texts are based on the run-length compressed Burrows-Wheeler transform (BWT) and on the Compact Directed Acyclic Word Graph (CDAWG). The most space-efficient indexes, on the other hand, are based on the Lempel-Ziv parsing and on grammar compression. Indexes for more universal schemes such as collage systems and macro schemes have not yet been proposed. Very recently, Kempa and Prezza [STOC 2018] showed that all dictionary compressors can be interpreted as approximation algorithms for the smallest string attractor, that is, a set of text positions capturing all distinct substrings. Starting from this observation, in this paper we develop the first universal compressed self-index, that is, the first indexing data structure based on string attractors, which can therefore be built on top of any dictionary-compressed text representation. Let γ be the size of a string attractor for a text of length n. From known reductions, γ can be chosen to be asymptotically equal to any repetitiveness measure: number of runs in the BWT, size of the CDAWG, number of Lempel-Ziv phrases, number of rules in a grammar or collage system, size of a macro scheme. Our index takes O(γ lg(n/γ)) words of space and supports locating the occ occurrences of any pattern of length m in O(m lg n + occ lg n) time, for any constant > 0. This is, in particular, the first index for general macro schemes and collage systems. Our result shows that the relation between indexing and compression is much deeper than what was previously thought: the simple property standing at the core of all dictionary compressors is sufficient to support fast indexed queries.

show abstract

Section: Building the Multisets X And Ymentioning

confidence: 99%

Universal compressed text indexing

Navarro

Prezza

2019

Theoretical Computer Science

View full text Add to dashboard Cite

show abstract

“…• LZEnd [45]. An algorithm to compute LZ-End parsing in O(n log ) time with high probability and O((z e + ) lg n) bits of space, where z e is the number of phrases of LZ-End and is the maximum length of the phrase.…”

Section: Resultsmentioning

confidence: 99%

“…We would like to thank the anonymous reviewers for their insightful comments to improve our manuscript. We also thank Dominik Kempa for sending us the implementation of [45].…”

Section: Acknowledgementsmentioning

confidence: 99%

Refining the r-index

Bannai

Gagie

Tomohiro

2020

Theoretical Computer Science

View full text Add to dashboard Cite

“…We note that the algorithmic techniques we have developed here can also be applied to, e.g., develop more space-efficient parsing algorithms for LZ-End [13], a variant of LZ77 [18] with which each phrase s[i..j] is the longest prefix of s[i..n] such that an occurrence of s[i..j−1] in s[1..i−1] ends at a phrase boundary. Kempa and Kosolobov [12] very recently gave an LZ-End parsing algorithm that runs in O(n log ℓ) expected time and O(z + ℓ) space, where ℓ is the length of the longest phrase and z is the number of phrases.…”

Section: Discussionmentioning

confidence: 99%

On Two LZ78-style Grammars: Compression Bounds and Compressed-Space Computation

Badkobeh¹,

Gagie²,

Inenaga³

et al. 2017

String Processing and Information Retrieval

Self Cite

View full text Add to dashboard Cite

We investigate two closely related LZ78-based compression schemes: LZMW (an old scheme by Miller and Wegman) and LZD (a recent variant by Goto et al.). Both LZD and LZMW naturally produce a grammar for a string of length n; we show that the size of this grammar can be larger than the size of the smallest grammar by a factor Ω(n 1 3 ) but is always within a factor O(( n log n ) 2 3 ). In addition, we show that the standard algorithms using Θ(z) working space to construct the LZD and LZMW parsings, where z is the size of the parsing, work in Ω(n 5 4 ) time in the worst case. We then describe a new Las Vegas LZD/LZMW parsing algorithm that uses O(z log n) space and O(n + z log 2 n) time with high probability.of the smallest grammar exists. This paper addresses that need. Moreover, we show that the standard algorithms for computing LZD and LZMW have undesirable worst case performance, and provide an alternative algorithm that runs in log-linear randomized time. In particular the contributions of this article are as follows:1. We show that the size of the grammar produced by LZD and LZMW can be larger than the size of the smallest grammar by a factor Ω(n 1 3 ) but is always within a factor O(( n log n ) 2 3 ). To our knowledge these are the first non-trivial bounds on compression performance known for these algorithms.2. Space usage during compression is often a concern. For both LZD and LZMW, parsing algorithms are known that use O(z) space, where z is the size of the final parsing. We describe strings for which these algorithms require Ω(n 5 4 ) time. (The only previous analysis is an O(n 2 / log n) upper bound [8].)3. We describe a Monte-Carlo parsing algorithm for LZD/LZMW that uses a z-fast trie [2] and an AVL-grammar [15] to achieve O(z log n) space and O(n + z log 2 n) time for inputs over the integer alphabet {0, 1, . . . , n O(1) }. This algorithm works in the streaming model and computes the parsing with high probability. Using the Monte-Carlo solution, we obtain a Las Vegas algorithm that, with high probability, works in the same space and time.In what follows we provide formal definitions and examples of LZD and LZMW parsings. Section 2 then establishes bounds for the approximation ratios for the sizes of the LZD/LZMW grammars. In Section 3 we consider the time efficiency of current space-efficient parsing schemes for LZD/LZMW. Section 4 provides an algorithm with significantly better (albeit randomized) performance. Conclusions and reflections are offered in Section 5.

show abstract

LZ-End Parsing in Compressed Space

Cited by 17 publications

References 15 publications

Universal compressed text indexing

Universal compressed text indexing

Refining the r-index

On Two LZ78-style Grammars: Compression Bounds and Compressed-Space Computation

Contact Info

Product

Resources

About