Arbitrary source models and Bayesian codebooks in rate-distortion theory

Abstract

Abstract-We characterize the best achievable performance of lossy compression algorithms operating on arbitrary random sources, and with respect to general distortion measures. Direct and converse coding theorems are given for variable-rate codes operating at a fixed distortion level, emphasizing: a) nonasymptotic results, b) optimal or near-optimal redundancy bounds, and c) results with probability one. This development is based in part on the observation that there is a precise correspondence between compression algorithms and probability measures on the reproduction alphabet. This is analogous to the Kraft inequality in lossless data compression. In the case of stationary ergodic sources our results reduce to the classical coding theorems. As an application of these general results, we examine the performance of codes based on mixture codebooks for discrete memoryless sources. A mixture codebook (or Bayesian codebook) is a random codebook generated from a mixture over some class of reproduction distributions. We demonstrate the existence of universal mixture codebooks, and show that it is possible to universally encode memoryless sources with redundancy of approximately ( 2) log bits, where is the dimension of the simplex of probability distributions on the reproduction alphabet.

Cumulant Generating Function of Codeword Lengths in Variable-Length Lossy Compression Allowing Positive Excess Distortion Probability

“…In our non-asymptotic analysis, the distortion D-ball around y (i.e., (26)) plays a crucial role. On the other hand, in the previous studies of non-asymptotics for lossy compression [3], [5], [6], [7], [8], the distortion D-ball around x (i.e.,B D (x) := {y ∈ Y : d(x, y) ≤ D}) plays an important role. Investigating the relation of approaches between previous works and our work is one of the future works.…”

Variable-length lossy compression allowing positive overflow and excess distortion probabilities

“…For the problem of variable-length lossy source coding under the excess distortion probability, there are several criteria on codeword lengths: for example, the mean codeword lengths which is discussed in [12]; the pointwise redundancy rate in [7], [8]; the cumulant generating function of codeword lengths in [1], [20]; the excess-codelength exponent in [25]; and the overflow probability in [14], [26]. Among them, Nomura and Yagi [14] and Yagi and Nomura [26] adopt the same criteria in this paper: the overflow probability and the excess distortion probability.…”

“…It should be noted that the distortion D-ball in the previous studies and the distortion D-ball in our study are different. In the previous studies (e.g., [7], [8], [10], [12]), the distortion D-ball centered at x, i.e., the quantity…”

“…• We define y i (i = 1, 2, · · · ) by the following procedure 8 . Let y 1 be defined as 7 Strictly speaking, we encode a source symbol x ∈ A D (y k * ) differently whether we use stochastic encoders or deterministic enoders.…”

“…In this case, to make the overflow probability smaller than δ, we must set the rate G δ D,ǫ (X) + 2 log e/2 G δ D,ǫ (X) , which is slightly larger than ⌊G δ D,ǫ (X)⌋. 8 In this paper, we assume that X and Y are finite sets. However, we can assume countably infinite X and Y if this operation is admitted for countably infinite X and Y. and for i = 2, 3, · · · , let y i be defined as…”