Lempel-Ziv Parsing in External Memory

Kärkkäinen, Juha; Kempa, Dominik; Puglisi, Simon J.

doi:10.1109/dcc.2014.78

Cited by 22 publications

(17 citation statements)

References 27 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Goto and Bannai [2013] utilized SACA-K to design a space-efficient linear-time algorithm for computing LZ77 factorization on constant alphabets. Kärkkäinen et al [2014] proposed algorithms for computing the LZ77 parsing efficiently using the external memory. This suggests a possibility for extending DSA-IS for computing LZ77 factorization in the external memory.…”

Section: Discussionmentioning

confidence: 99%

Induced Sorting Suffixes in External Memory

Nong

Chan

et al. 2015

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

We present in this article an external memory algorithm, called disk SA-IS (DSA-IS), to exactly emulate the induced sorting algorithm SA-IS previously proposed for sorting suffixes in RAM. DSA-IS is a new diskfriendly method for sequentially retrieving the preceding character of a sorted suffix to induce the order of the preceding suffix. For a size-n string of a constant or integer alphabet, given the RAM capacity ((nW ) 0.5 ), where W is the size of each I/O buffer that is large enough to amortize the overhead of each access to disk, both the CPU time and peak disk use of DSA-IS are O(n). Our experimental study shows that on average, DSA-IS achieves the best time and space results of all of the existing external memory algorithms based on the induced sorting principle.

show abstract

Section: Discussionmentioning

confidence: 99%

Induced Sorting Suffixes in External Memory

Nong

Chan

et al. 2015

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…Recently, in an attempt to address this standoff, Ferrada et al [3] described hybrid indexing -an algorithmic technique by which any conventional pattern matching index (including any read aligner) can be made to scale to large, highly compressible collections via means of the Lempel-Ziv (LZ77) parsing [30,14,11], a method from data compression (we give a formal definition shortly). In particular, given an upper bound M on the searchable pattern length, the first step of hybrid indexing is to obtain a filtered string consisting of the concatenation of the M -length substrings to the left and right of each LZ77 phrase boundary.…”

Section: Introductionmentioning

confidence: 99%

Hybrid Indexing Revisited

Ferrada

Kempa

Puglisi

2018

2018 Proceedings of the Twentieth Workshop on Algorithm Engineering and Experiments (ALENEX)

Self Cite

View full text Add to dashboard Cite

Hybrid indexing is a recent approach to text indexing that allows the space-usage of conventional text indexes (e.g., suffix trees, suffix arrays, FM-indexes) to scale well with the text size, n, when z, the size of the Lempel-Ziv parsing of the text, is small relative to n. The price for this improved scalability is that an upper bound M on the pattern length that can be searched for must be declared at index construction time. Because the size of the resulting index contains an O(M z) term, M must be kept reasonably small, though it has been shown that M ≈ 100 leads to acceptable performance in some genomic applications. However, despite its promise, the practical performance of hybrid indexing relative to other compressed index data structures is poorly understood. This paper addresses that need, detailing experiments that show hybrid indexing -when carefully implemented -to be significantly smaller and faster than alternative approaches on a broad range of data of different levels of compressibility. We also describe practical extensions to hybrid indexing that obviate the restriction on M , supporting search for patterns of arbitrary length.

show abstract

“…It is also used for genome assembly [3,2,1] and extensively for the discovery of repetitive structures in genomic data [22]. Elsewhere, in data compression, it is the index underlying state-of-the-art methods for Lempel-Ziv factorization [20,18]. The key virtue of the index, in these and other applications, is that it stores a string T in a compressed form that also supports fast pattern matching queries.…”

Section: Introductionmentioning

confidence: 99%

Fixed Block Compression Boosting in FM-Indexes: Theory and Practice

et al. 2018

Self Cite

View full text Add to dashboard Cite

The FM index (Ferragina & Manzini, J. ACM, 2005) is a widely-used compressed data structure that stores a string T in a compressed form and also supports fast pattern matching queries. In this paper, we describe new FM-index variants that combine nice theoretical properties, simple implementation and improved practical performance. Our main theoretical result is a new technique called fixed block compression boosting, which is a simpler and faster alternative to optimal compression boosting and implicit compression boosting used in previous FM-indexes. We also describe several new techniques for implementing fixed-block boosting efficiently, including a new, fast, and space-efficient implementation of wavelet trees. Our extensive experiments show the new indexes to be consistently fast and small relative to the state-of-the-art, and thus they make a good "off-the-shelf" choice for many applications.

show abstract

Lempel-Ziv Parsing in External Memory

Cited by 22 publications

References 27 publications

Induced Sorting Suffixes in External Memory

Induced Sorting Suffixes in External Memory

Hybrid Indexing Revisited

Fixed Block Compression Boosting in FM-Indexes: Theory and Practice

Contact Info

Product

Resources

About