Simple Compression Code Supporting Random Access and Fast String Matching

Abstract

Abstract. Given a sequence S of n symbols over some alphabet Σ, we develop a new compression method that is (i) very simple to implement; (ii) provides O(1) time random access to any symbol of the original sequence; (iii) allows efficient pattern matching over the compressed sequence. Our simplest solution uses at most 2h + o(h) bits of space, where h = n(H 0 (S) + 1), and H 0 (S) is the zeroth-order empirical entropy of S. We discuss a number of improvements and trade-offs over the basic method. The new method is applied to text compression. We also propose average case optimal string matching algorithms.

Hash, Displace, and Compress

“…The compression scheme from [11] will thus give an expected space bound of n(log e + O((log(λ) + 1)/λ)) + O(λ 2 ) (the O-notation refers to growing λ), as shown in Appendix A.2. More sophisticated compression schemes for the index table T, which can reduce the effect of the fact that the code length must be an integer while log(σ(i)) is not, will be able to better approximate the optimum n log e.…”

“…The evaluation time of the PHFs and MPHFs generated by CHD algorithm depends on the compression technique used. For instance, it is possible to generate faster functions using Elias-Fano scheme (see [27]) instead of the one we used for the experiments [11] at the expense of generating functions with a slightly larger description size (we obtained PHFs that require 2.08 bits per key instead of 1.98 bits per key for α = 0.99 and λ = 5).…”

“…However, the hidden constants in the o(n) and O(1) notation may not be so good in practice. For a more practical solution we can use the method proposed by Fredriksson and Nikitin [11]. This solution is particularly interesting for our case because it does not need to use any encoding or decoding tables.…”

“…We will also show that the numbers σ(i), 0 ≤ i < r, are small, in that σ(i) is geometrically distributed with bounded expectation, so that E(σ(i)) is bounded by a constant. From this it follows that by using a compression scheme like that described in [11] the whole sequence (σ(i), 0 ≤ i < r) can be coded in O(n) bits, in a way that random access is possible, and so the property is retained that h can be evaluated in constant time.…”

“…As an estimate for this, we take ⌊log 2 (σ(i))⌋ + 1 the binary length of σ(i), which can be upper bounded by log 2 (σ(i)) + 1. (There are simple methods for storing these numbers using space 2 i<r (log 2 (σ(i))+ 1), that allow random access in O(1) time, given e. g. in [11].) Thus, 2r + O( i<r (log 2 (σ(i)))) is an upper bound for the space needed to store the PHF h. We let the random variable s denote i<r log 2 (σ(i)) and estimate E(s).…”