Fine-Grained Complexity of Analyzing Compressed Data: Quantifying Improvements over Decompress-and-Solve

Abstract

Can we analyze data without decompressing it? As our data keeps growing, understanding the time complexity of problems on compressed inputs, rather than in convenient uncompressed forms, becomes more and more relevant. Suppose we are given a compression of size n of data that originally has size N , and we want to solve a problem with time complexity T (·). The naïve strategy of "decompress-and-solve" gives time T (N ), whereas "the gold standard" is time T (n): to analyze the compression as efficiently as if the original data was small.We restrict our attention to data in the form of a string (text, files, genomes, etc.) and study the most ubiquitous tasks. While the challenge might seem to depend heavily on the specific compression scheme, most methods of practical relevance (Lempel-Ziv-family, dictionary methods, and others) can be unified under the elegant notion of Grammar-Compressions. A vast literature, across many disciplines, established this as an influential notion for Algorithm design.We introduce a framework for proving (conditional) lower bounds in this field, allowing us to assess whether decompress-and-solve can be improved, and by how much. Our main results are: • The O(nN log N/n) bound for LCS and the O(min{N log N, nM }) bound for Pattern Matching with Wildcards are optimal up to N o(1) factors, under the Strong Exponential Time Hypothesis. (Here, M denotes the uncompressed length of the compressed pattern.)• Decompress-and-solve is essentially optimal for Context-Free Grammar Parsing and RNA Folding, under the k-Clique conjecture.• We give an algorithm showing that decompress-and-solve is not optimal for Disjointness.

More consequences of falsifying SETH and the orthogonal vectors conjecture

“…We apply Lemma 3.4. In particular, setting p = √ log n, we get k dp = n o(1) queries and maximum weight M ≤ O(k 4d M 2/p p) = n o (1) .…”

“…, k} with 1 ≤ |C| ≤ d, we exhaustively guess a weight a C ∈ [−M , M ]. In total, this requires iterating through at most (2M + 1) k d = n o (1) candidate weight vectors a = (a C ) C⊆{1,...,k} . If the sum C a C is not equal to t , we reject the candidate vector and move to the next one.…”

“…is the output gate of C. C2 (Replace large threshold gates by the circuit in Figure 2 a Let C v,A be the subcircuit of C induced by the set R C (A, v) (see (1) in the preliminaries) and interpret the gates a ∈ A as input variables y a . b We will show that C v,A depends on at most β d variables, so we can compute a k-CNF formula F v with at most k = β d variables that expresses the constraint "y v = C v,A (x, y)", where x 1 , .…”

Regular Expression Search on Compressed Text

“…• Accessing the list of states q 2 with (q ′ , β ℓ , q 2 ) ∈ δ at line 16 (using N ). For non deterministic FSA, they proved [7,Thm. 4.2] that, under the k-Clique Conjecture, there is no combinatorial algorithm running in O((p·s 3 ) 1−ε ) time.…”

“…Indeed, zearch exhibits almost linear behavior with respect to the size of the FSA built from the expression. Nevertheless, there are regular expressions that trigger the worst case behavior (last row in Table 2), which cannot be avoided due to the result of Abboud et al [7] describe before.…”