Arbitrary source models and Bayesian codebooks in rate-distortion theory

Kontoyiannis, Ioannis; Zhang, Junshan

doi:10.1109/tit.2002.800493

Cited by 22 publications

(28 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the entropy H(q(X 1 )) < ∞. The following was implicitly proved in [43]; see also [51] for details. [43]: Let X be a stationary ergodic process.…”

Section: Generalized Aep For Optimal Lossy Compressionmentioning

confidence: 83%

“…The observations of Theorem 4 about the rate-function R(P, Q, D) are new. Theorem 5 essentially comes from Kieffer's work [43]; see also [51].…”

Section: Theorem 5 Generalized Aep For Optimal Lossy Compressionmentioning

confidence: 99%

“…We remark that the coefficients of the (log n) terms in (a) and (b) above are not the best possible, and can be significantly improved; see [51] for more details.…”

Section: Random Codes and Second Order Conversesmentioning

confidence: 99%

See 2 more Smart Citations

Source coding, large deviations, and approximate pattern matching

Dembo

Kontoyiannis

2002

IEEE Trans. Inform. Theory

107

View full text Add to dashboard Cite

-We present a development of parts of rate-distortion theory and pattern-matching algorithms for lossy data compression, centered around a lossy version of the Asymptotic Equipartition Property (AEP). This treatment closely parallels the corresponding development in lossless compression, a point of view that was advanced in an important paper of Wyner and Ziv in 1989. In the lossless case we review how the AEP underlies the analysis of the Lempel-Ziv algorithm by viewing it as a random code and reducing it to the idealized Shannon code. This also provides information about the redundancy of the Lempel-Ziv algorithm and about the asymptotic behavior of several relevant quantities.In the lossy case we give various versions of the statement of the generalized AEP and we outline the general methodology of its proof via large deviations. Its relationship with Barron and Orey's generalized AEP is also discussed. The lossy AEP is applied to: (i) prove strengthened versions of Shannon's direct source coding theorem and universal coding theorems; (ii) characterize the performance of "mismatched" codebooks in lossy data compression; (iii) analyze the performance of pattern-matching algorithms for lossy compression (including Lempel-Ziv schemes); (iv) determine the first order asymptotics of waiting times (with distortion) between stationary processes; (v) characterize the best achievable rate of "weighted" codebooks as an optimal sphere-covering exponent. We then present a refinement to the lossy AEP and use it to: (i) prove second order (direct and converse) lossy source coding theorems, including universal coding theorems; (ii) characterize which sources are quantitatively easier to compress; (iii) determine the second order asymptotics of waiting times between stationary processes; (iv) determine the precise asymptotic behavior of longest match-lengths between stationary processes. Extensions to random fields are also given.

show abstract

“…the entropy H(q(X 1 )) < ∞. The following was implicitly proved in [43]; see also [51] for details. [43]: Let X be a stationary ergodic process.…”

Section: Generalized Aep For Optimal Lossy Compressionmentioning

confidence: 83%

“…The observations of Theorem 4 about the rate-function R(P, Q, D) are new. Theorem 5 essentially comes from Kieffer's work [43]; see also [51].…”

Section: Theorem 5 Generalized Aep For Optimal Lossy Compressionmentioning

confidence: 99%

See 1 more Smart Citation

Source coding, large deviations, and approximate pattern matching

Dembo

Kontoyiannis

2002

IEEE Trans. Inform. Theory

107

View full text Add to dashboard Cite

show abstract

“…Step 4: expected first-stage description length. We follow the ideas of [10]. Let us assume that the sequence…”

Section: The Proof Of Theoremmentioning

confidence: 99%

Joint Universal Lossy Coding and Identification of Stationary Mixing Sources

Raginsky

2007

2007 IEEE International Symposium on Information Theory

View full text Add to dashboard Cite

The problem of joint universal source coding and modeling, treated in the context of lossless codes by Rissanen, was recently generalized to fixed-rate lossy coding of finitely parametrized continuous-alphabet i.i.d. sources. We extend these results to variable-rate lossy block coding of stationary ergodic sources and show that, for bounded metric distortion measures, any finitely parametrized family of stationary sources satisfying suitable mixing, smoothness and Vapnik-Chervonenkis learnability conditions admits universal schemes for joint lossy source coding and identification. We also give several explicit examples of parametric sources satisfying the regularity conditions.

show abstract

“…In order to ensure that such a description can be given with finite rate, we introduce the following simple conditions; cf. [17,19,8]. (pSQC): For a distortion level D ≥ 0 we say that the p-strong quantization condition (pSQC) holds at D for some p ≥ 1, if (WQC) holds with respect to a scalar quantizer q also satisfying…”

Section: Naive Codingmentioning

confidence: 99%

Mismatched codebooks and the role of entropy-coding in lossy data compression

Kontoyiannis

Zamir

2003

IEEE International Symposium on Information Theory, 2003. Proceedings.

Self Cite

View full text Add to dashboard Cite

-We introduce a universal quantization scheme based on random coding, and we analyze its performance. This scheme consists of a source-independent random codebook (typically mismatched to the source distribution), followed by optimal entropy-coding that is matched to the quantized codeword distribution. A single-letter formula is derived for the rate achieved by this scheme at a given distortion, in the limit of large codebook dimension. The rate reduction due to entropycoding is quantified, and it is shown that it can be arbitrarily large. In the special case of "almost uniform" codebooks (e.g., an i.i.d. Gaussian codebook with large variance) and difference distortion measures, a novel connection is drawn between the compression achieved by the present scheme and the performance of "universal" entropy-coded dithered lattice quantizers. This connection generalizes the "half-a-bit" bound on the redundancy of dithered lattice quantizers. Moreover, it demonstrates a strong notion of universality where a single "almost uniform" codebook is near-optimal for any source and any difference distortion measure. The proofs are based on the fact that the limiting empirical distribution of the first matching codeword in a random codebook can be precisely identified. This is done using elaborate large deviations techniques, that allow the derivation of a new "almost sure" version of the conditional limit theorem.

show abstract

Arbitrary source models and Bayesian codebooks in rate-distortion theory

Cited by 22 publications

References 26 publications

Source coding, large deviations, and approximate pattern matching

Source coding, large deviations, and approximate pattern matching

Joint Universal Lossy Coding and Identification of Stationary Mixing Sources

Mismatched codebooks and the role of entropy-coding in lossy data compression

Contact Info

Product

Resources

About