Hybrid Indexing Revisited

Ferrada, Héctor; Kempa, Dominik; Puglisi, Simon J.

doi:10.1137/1.9781611975055.1

Cited by 14 publications

(10 citation statements)

References 21 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared r-index with the state-of-the-art index for each compressibility measure: lzi 23 [73,24] (z), slp 23 [25,24] (g), rlcsa 24 [79,80] (r), and cdawg 25 [9] (e). We also included hyb 26 [30,31], which combines a Lempel-Ziv index with an FM-index, with parameter M = 8, which is optimal for our experiment. We tested rlcsa using three suffix array sample rates per dataset: the rate X resulting in the same size for rlcsa and r-index, plus rates X/2 and X/4.…”

Section: Methodsmentioning

confidence: 99%

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

123

142

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transforms (BWTs). One of the earliest indexes for repetitive collections, the Run-Length FM-index, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitivenessthe number of phrases in the Lempel-Ziv parse, the size of the smallest grammar generating (only) the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FM-index so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time, O(m + occ), within O(r log log w (σ + n/r)) space, for a text of length n over an alphabet of size σ on a RAM machine with words of w = Ω(log n) bits. Within that space, our index can also count in optimal time, O(m). Multiplying the space by O(w/ log σ), we support count and locate in O( m log(σ)/w ) and O( m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and extracts any text substring of length in almost-optimal time O(log(n/r) + log(σ)/w). Within that space, we similarly provide direct access to suffix array, inverse suffix array, and longest common prefix array cells, and extend these capabilities to full suffix tree functionality, typically in O(log(n/r)) time per operation. Our experiments show that our O(r)-space index outperforms the space-competitive alternatives by 1-2 orders of magnitude.

show abstract

Section: Methodsmentioning

confidence: 99%

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

123

142

View full text Add to dashboard Cite

show abstract

“…5 We use Patricia trees sampling values s = 4, 16, 64. lz-index and lze-index: Two variants of the Lempel-Ziv based index [15]. 6 hyb-index: A hybrid between a Lempel-Ziv and a BWT-based index [7]. 7 We build it with parameters M = 8, 16, the best for this case.…”

Section: Tested Indexesmentioning

confidence: 99%

“…6 hyb-index: A hybrid between a Lempel-Ziv and a BWT-based index [7]. 7 We build it with parameters M = 8, 16, the best for this case.…”

Section: Tested Indexesmentioning

confidence: 99%

See 1 more Smart Citation

A Fast and Small Subsampled R-index

Cobas,

Gagie,

Navarro

2021

Preprint

View full text Add to dashboard Cite

The r -index (Gagie et al., JACM 2020) represented a breakthrough in compressed indexing of repetitive text collections, outperforming its alternatives by orders of magnitude. Its space usage, O(r) where r is the number of runs in the Burrows-Wheeler Transform of the text, is however larger than Lempel-Ziv and grammar-based indexes, and makes it uninteresting in various real-life scenarios of milder repetitiveness. In this paper we introduce the sr -index, a variant that limits the space to O(min(r, n/s)) for a text of length n and a given parameter s, at the expense of multiplying by s the time per occurrence reported. The sr -index is obtained by carefully subsampling the text positions indexed by the r -index, in a way that we prove is still able to support pattern matching with guaranteed performance. Our experiments demonstrate that the sr -index sharply outperforms virtually every other compressed index on repetitive texts, both in time and space, even matching the performance of the r -index while using 1.5-3.0 times less space. Only some Lempel-Ziv-based indexes achieve better compression than the sr -index, using about half the space, but they are an order of magnitude slower.

show abstract

“…This has a major impact on the final size of the index. For this reason, the hybrid index is suitable mainly for short-read alignment, although there have been recent heuristic modifications to allow for longer alignments (Ferrada et al, 2018). In contrast, the r-index, of which we provide an implementation in this work, has no such length limitation.…”

Section: Related Workmentioning

confidence: 99%

Efficient Construction of a Complete Index for Pan-Genomics Read Alignment

Kuhnle

Mun²,

Boucher

et al. 2020

Journal of Computational Biology

View full text Add to dashboard Cite

Short-read aligners predominantly use the FM-index, which is easily able to index one or a few human genomes. However, it does not scale well to indexing collections of thousands of genomes. Driving this issue are the two chief components of the index: (1) a rank data structure over the Burrows-Wheeler Transform (BWT) of the string that will allow us to find the interval in the string's suffix array (SA), and (2) a sample of the SA that-when used with the rank data structure-allows us to access the SA. The rank data structure can be kept small even for large genomic databases, by run-length compressing the BWT, but until recently there was no means known to keep the SA sample small without greatly slowing down access to the SA. Now that (SODA 2018) has defined an SA sample that takes about the same space as the run-length compressed BWT, we have the design for efficient FM-indexes of genomic databases but are faced with the problem of building them. In 2018, we showed how to build the BWT of large genomic databases efficiently (WABI 2018), but the problem of building the sample efficiently was left open. We compare our approach to state-of-the-art methods for constructing the SA sample, and demonstrate that it is the fastest and most space-efficient method on highly repetitive genomic databases. Lastly, we apply our method for indexing partial and whole human genomes and show that it improves over the FMindex-based Bowtie method with respect to both memory and time and over the hybrid index-based CHIC method with respect to query time and memory required for indexing.

show abstract

Hybrid Indexing Revisited

Cited by 14 publications

References 21 publications

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

A Fast and Small Subsampled R-index

Efficient Construction of a Complete Index for Pan-Genomics Read Alignment

Contact Info

Product

Resources

About