On empirical cumulant generating functions of code lengths for individual sequences

Abstract

We consider the problem of lossless compression of individual sequences using finite-state (FS) machines, from the perspective of the best achievable empirical cumulant generating function (CGF) of the code length, i.e., the normalized logarithm of the empirical average of the exponentiated code length. Since the probabilistic CGF is minimized in terms of the Rényi entropy of the source, one of the motivations of this study is to derive an individual-sequence analogue of the Rényi entropy, in the same way that the FS compressibility is the individual-sequence counterpart of the Shannon entropy. We consider the CGF of the code-length both from the perspective of fixed-to-variable (F-V) length coding and the perspective of variable-to-variable (V-V) length coding, where the latter turns out to yield a better result, that coincides with the FS compressibility. We also extend our results to compression with side information, available at both the encoder and decoder. In this case, the V-V version no longer coincides with the FS compressibility, but results in a different complexity measure.

Improved Bounds on Lossless Source Coding and Guessing Moments via Rényi Measures

“…These bounds, expressed in terms of the Rényi entropy, imply that for sufficiently long source sequences, it is possible to make the normalized cumulant generating function of the codeword lengths approach the Rényi entropy as closely as desired by a proper fixed-to-variable uniquely-decodable source code; moreover, a converse result in [13] shows that there is no uniquely-decodable source code for which the normalized cumulant generating function of its codeword lengths lies below the Rényi entropy. In addition, this type of bounds was studied in the context of various coding problems, including guessing (see, e.g., [1], [2], [3], [7], [8], [9], [16], [17], [22], [28], [33], [34], [35], [46], [50]).…”