Reducing the Space Requirement of LZ-Index

Arroyuelo, Diego; Navarro, Gonzalo; Sadakane, Kunihiko

doi:10.1007/11780441_29

Cited by 28 publications

(50 citation statements)

References 21 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We plan to try out others that have mild dependence on the alphabet size. In particular, adapting the LZ-index [23,2] should offer fast locating of occurrences.…”

Section: Discussionmentioning

confidence: 99%

Self-indexing Natural Language

Brisaboa

Fariña

Navarro

et al. 2008

String Processing and Information Retrieval

Self Cite

View full text Add to dashboard Cite

Abstract. Self-indexing is a concept developed for indexing arbitrary strings. It has been enormously successful to reduce the size of the large indexes typically used on strings, namely suffix trees and arrays. Selfindexes represent a string in a space close to its compressed size and provide indexed searching on it. On natural language, a compressed inverted index over the compressed text already provides a reasonable alternative, in space and time, for indexed searching of words and phrases. In this paper we explore the possibility of regarding natural language text as a string of words and applying a self-index to it. There are several challenges involved, such as dealing with a very large alphabet and detaching searchable content from non-searchable presentation aspects in the text. As a result, we show that the self-index requires space very close to that of the best word-based compressors, and that it obtains better search time than inverted indexes (using the same overall space) when searching for phrases.

show abstract

“…We plan to try out others that have mild dependence on the alphabet size. In particular, adapting the LZ-index [23,2] should offer fast locating of occurrences.…”

Section: Discussionmentioning

confidence: 99%

Self-indexing Natural Language

Brisaboa

Fariña

Navarro

et al. 2008

String Processing and Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…The case when P overlaps three or more words is a generalization of the case with two words. Recently, Arroyuelo et al [1] improved the original data structure in [10]. For any text T,t h e improved index uses (2 + )jTjH k (T)+o(jTj log j˙j)bits of space, where H k (T)i st h ek-th order empirical entropy of T, and reports all occ occurrences of P in T in O(jPj 2 log jPj +(jPj + occ)logjTj)time.…”

Section: Compressed Full-text Indexingmentioning

confidence: 99%

Dictionary-Based Data Compression

Gagie¹,

Manzini²

2016

Encyclopedia of Algorithms

View full text Add to dashboard Cite

show abstract

“…A new LZ-index, the Inverted LZ-index (ILZI for short) [38], has appeared independently and simultaneously with our work [4]. The ILZI is faster than our data structures since it can report the pattern occurrences in O((m + occ) log u) time, but at the price of a higher space requirement: (5 + ǫ)uH k (T ) + o(u log σ) bits.…”

Section: Our Contributionmentioning

confidence: 99%

Stronger Lempel-Ziv Based Compressed Text Indexing

Arroyuelo¹,

Navarro

Sadakane

2010

Algorithmica

Self Cite

View full text Add to dashboard Cite

Abstract. Given a text T [1..u] over an alphabet of size σ, the full-text search problem consists in finding the occ occurrences of a given pattern P [1..m] in T . In indexed text searching we build an index on T to improve the search time, yet increasing the space requirement. The current trend in indexed text searching is that of compressed full-text self-indices, which replace the text with a more space-efficient representation of it, at the same time providing indexed access to the text. Thus, we can provide efficient access within compressed space. The LZ-index of Navarro is a compressed full-text self-index able to represent T using 4uH k (T ) + o(u log σ) bits of space, where H k (T ) denotes the k-th order empirical entropy of T , for any k = o(log σ u). This space is about four times the compressed text size. It can locate all the occ occurrences of a pattern P in T in O(m 3 log σ + (m + occ) log u) worst-case time. Despite this index has shown to be very competitive in practice, the O(m 3 log σ) term can be excessive for long patterns. Also, the factor 4 in its space complexity makes it larger than other state-of-the-art alternatives. In this paper we present stronger Lempel-Ziv based indices, improving the overall performance of the LZ-index. We achieve indices requiring (2 + ǫ)uH k (T ) + o(u log σ) bits of space, for any constant ǫ > 0, which makes our indices the smallest existing LZ-indices. We simultaneously improve the search time to O(m 2 +(m+occ) log u), which makes our indices very competitive with state-of-the-art alternatives. Our indices support displaying of any text substring of length ℓ in optimal O(ℓ/ log σ u) time. In addition, we show how the space can be squeezed to (1 + ǫ)uH k (T ) + o(u log σ) to obtain a structure with O(m 2 ) average search time for m 2 log σ u. Alternatively, the search time of LZ-indices can be improved to O((m + occ) log u) with (3 + ǫ)uH k (T ) + o(u log σ) bits of space, which is about half of the space needed by other Lempel-Ziv-based indices achieving the same search time. Overall our indices stand out as a very attractive alternative for space-efficient indexed text searching.

show abstract

Reducing the Space Requirement of LZ-Index

Cited by 28 publications

References 21 publications

Self-indexing Natural Language

Self-indexing Natural Language

Dictionary-Based Data Compression

Stronger Lempel-Ziv Based Compressed Text Indexing

Contact Info

Product

Resources

About