Indexed Matching Statistics and Shortest Unique Substrings

The field of succinct data structures has flourished over the last 16 years. Starting from the compressed suffix array by Grossi and Vitter (STOC 2000) and the FM-index by Ferragina and Manzini (FOCS 2000), a number of generalizations and applications of string indexes based on the Burrows-Wheeler transform (BWT) have been developed, all taking an amount of space that is close to the input size in bits. In many large-scale applications, the construction of the index and its usage need to be considered as one unit of computation. For example, one can compare two genomes by building a common index for their concatenation, and by detecting common substructures by querying the index. Efficient string indexing and analysis in small space lies also at the core of a number of primitives in the data-intensive field of high-throughput DNA sequencing.We report the following advances in string indexing and analysis. We show that the BWT of a string T ∈ {1, . . . , σ} n can be built in deterministic O(n) time using just O(n log σ) bits of space, where σ ≤ n. Deterministic linear time is achieved by exploiting a new partial rank data structure that supports queries in constant time, and that might have independent interest. Within the same time and space budget, we can build an index based on the BWT that allows one to enumerate all the internal nodes of the suffix tree of T . Many fundamental string analysis problems, such as maximal repeats, maximal unique matches, and string kernels, can be mapped to such enumeration, and can thus be solved in deterministic O(n) time and in O(n log σ) bits of space from the input string, by tailoring the enumeration algorithm to some problem-specific computations.We also show how to build many of the existing indexes based on the BWT, such as the compressed suffix array, the compressed suffix tree, and the bidirectional BWT index, in randomized O(n) time and in O(n log σ) bits of space. The previously fastest construction algorithms for BWT, compressed suffix array and compressed suffix tree, which used O(n log σ) bits of space, took O(n log log σ) time for the first two structures, and O(n log ǫ n) time for the third, where ǫ is any positive constant smaller than one. Contrary to the state of the art, our bidirectional BWT index supports every operation in constant time per element in its output.

show abstract

“…. This property allows all three of these vectors to be encoded in 2x bits, where x is the length of the corresponding input string [62,8].…”

Section: Matching Statisticsmentioning

confidence: 99%

Linear-time String Indexing and Analysis in Small Space

Belazzougui

Cunial

Kärkkäinen

et al. 2020

ACM Trans. Algorithms

Self Cite

View full text Add to dashboard Cite

show abstract

“…This computation can be done in O(|y|) time [2,24]. By applying Fact 1, we can answer any query y in O(|y|) time for…”

Section: Square-free-preserved Matching Statisticsmentioning

confidence: 99%

Longest property-preserved common factor: A new string-processing framework

Ayad

Bernardini

Grossi

et al. 2020

Theoretical Computer Science

View full text Add to dashboard Cite

a r t i c l e i n f o a b s t r a c tWe introduce a new family of string processing problems. Given two or more strings, we are asked to compute a factor common to all strings that preserves a specific property and has maximal length. We consider three fundamental string properties: square-free factors, periodic factors, and palindromic factors under three different settings, one per property. In the first setting, we are given a string x and we are asked to construct a data structure over x answering the following type of online queries: given a string y, find a longest squarefree factor common to x and y. In the second setting, we are given k strings and an integer 1 < k ≤ k and we are asked to find a longest periodic factor common to at least k strings.In the third one, we are given two strings and we are asked to find a longest palindromic factor common to the two strings. We present linear-time solutions for all settings. This is a full and extended version of a paper from SPIRE 2018.

show abstract

“…Computing MS S,T is a classical problem in string processing, and in practice it involves building an index on a fixed T to answer a large number of queries S. Thus, solutions typically differ on the index they use, which can be the textbook suffix tree, the compressed suffix tree [29] or compressed suffix array, the colored longest common prefix array [17], a Burrows-Wheeler index combined with the suffix tree topology [3,4], or the r-index combined with balanced grammars [6]. In the frequent case where T consists of one genome (or proteome), or of the concatenation of few similar genomes or of many dissimilar genomes, the Burrows-Wheeler transform of T does not compress well, and the best space-time tradeoffs are achieved by the implementation in [4] (see [6] for a runtime comparison, and see Figure 2 in the supplement for a memory comparison).…”

Section: Introductionmentioning

confidence: 99%

“…Then, we implement fast range queries for computing the average and maximum matching statistic value inside a substring of S, taking advantage of the compact encoding of MS S,T introduced by [3]: this encoding takes just 2|S| bits, and allows one to retrieve MS[i] in constant time for any i using just o(|S|) more bits. In some cases this bitvector is compressible, so our code can operate both on the plain encoding and on its compressed versions.…”

Section: Introductionmentioning

confidence: 99%

Fast and compact matching statistics analytics

Cunial

Denas²,

Belazzougui

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Fast, lightweight methods for comparing the sequence of ever larger assembled genomes from ever growing databases are increasingly needed in the era of accurate long reads and pan-genome initiatives. Matching statistics is a popular method for computing whole-genome phylogenies and for detecting structural rearrangements between two genomes, since it is amenable to fast implementations that require a minimal setup of data structures. However, current implementations use a single core, take too much memory to represent the result, and do not provide efficient ways to analyze the output in order to explore local similarities between the sequences. We develop practical tools for computing matching statistics between large-scale strings, and for analyzing its values, faster and using less memory than the state of the art. Specifically, we design a parallel algorithm for shared-memory machines that computes matching statistics 30 times faster with 48 cores in the cases that are most difficult to parallelize. We design a lossy compression scheme that shrinks the matching statistics array to a bitvector that takes from 0.8 to 0.2 bits per character, depending on the dataset and on the value of a threshold, and that achieves 0.04 bits per character in some variants. And we provide efficient implementations of range-maximum and range-sum queries that take a few tens of milliseconds while operating on our compact representations, and that allow computing key local statistics about the similarity between two strings. Our toolkit makes construction, storage, and analysis of matching statistics arrays practical for multiple pairs of the largest genomes available today, possibly enabling new applications in comparative genomics.

show abstract

Indexed Matching Statistics and Shortest Unique Substrings

Cited by 13 publications

References 22 publications

Linear-time String Indexing and Analysis in Small Space

Linear-time String Indexing and Analysis in Small Space

Longest property-preserved common factor: A new string-processing framework

Fast and compact matching statistics analytics

Contact Info

Product

Resources

About