Source coding, large deviations, and approximate pattern matching

Abstract

-We present a development of parts of rate-distortion theory and pattern-matching algorithms for lossy data compression, centered around a lossy version of the Asymptotic Equipartition Property (AEP). This treatment closely parallels the corresponding development in lossless compression, a point of view that was advanced in an important paper of Wyner and Ziv in 1989. In the lossless case we review how the AEP underlies the analysis of the Lempel-Ziv algorithm by viewing it as a random code and reducing it to the idealized Shannon code. This also provides information about the redundancy of the Lempel-Ziv algorithm and about the asymptotic behavior of several relevant quantities.In the lossy case we give various versions of the statement of the generalized AEP and we outline the general methodology of its proof via large deviations. Its relationship with Barron and Orey's generalized AEP is also discussed. The lossy AEP is applied to: (i) prove strengthened versions of Shannon's direct source coding theorem and universal coding theorems; (ii) characterize the performance of "mismatched" codebooks in lossy data compression; (iii) analyze the performance of pattern-matching algorithms for lossy compression (including Lempel-Ziv schemes); (iv) determine the first order asymptotics of waiting times (with distortion) between stationary processes; (v) characterize the best achievable rate of "weighted" codebooks as an optimal sphere-covering exponent. We then present a refinement to the lossy AEP and use it to: (i) prove second order (direct and converse) lossy source coding theorems, including universal coding theorems; (ii) characterize which sources are quantitatively easier to compress; (iii) determine the second order asymptotics of waiting times between stationary processes; (iv) determine the precise asymptotic behavior of longest match-lengths between stationary processes. Extensions to random fields are also given.

Lossy Asymptotic Equipartition Property for Networked Data Structures

“…We present and prove process Large Deviation Principle (LDP) for the coloured random graph conditioned to have a given empirical colour measure and empirical pair measure, see Doku-Amponsah (2006), using similar coupling techniques as in the article by Boucheron et al (2002). From this LDP and the techniques employed by Dembo and Kontoyiannis (2002) for the random field on Z 2 , we obtain the proof of the Lossy AEP for the Networked Data Structures.…”

Operational Duality Between Lossy Compression and Channel Coding

“…Informally, is the minimum rate required for a random codebook with entries chosen independently from a given so as to compress the source with average asymptotic distortion less than [18]. Note that (by substituting ).…”

Mismatched codebooks and the role of entropy-coding in lossy data compression

“…As mentioned in the Introduction, this coding scheme (and many variations on it) has been analyzed extensively in [28][29] [8] and several other works cited therein. To avoid potentially infinite searches in the codebook, we make the simplifying assumption that the encoder only describes N n when it is smaller than 2 nb , where b is some positive constant to be chosen later.…”