Improved Bounds on Lossless Source Coding and Guessing Moments via Rényi Measures

Abstract

This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probability are also established, characterizing the exact locus of their attainable values. The bounds on optimal guessing moments serve to improve non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints. Non-asymptotic bounds on the reliability function of discrete memoryless sources are derived as well. Relying on these techniques, lower bounds on the cumulant generating function of the codeword lengths are derived, by means of the smooth Rényi entropy, for source codes that allow decoding errors.

Tight Bounds on the Rényi Entropy via Majorization with Applications to Guessing and Compression

“…A. Guessing 1) Background: The problem of guessing discrete random variables has various theoretical and operational aspects in information theory (see [1], [2], [3], [10], [11], [14], [17], [31], [32], [41], [54], [55], [56], [59], [65], [68], [74], [75], [85]). The central object of interest is the distribution of the number of guesses required to identify a realization of a random variable X, taking values on a finite or countably infinite set X = {1, .…”

“…Not only does this strategy minimize the average number of guesses, but it also minimizes the ρ-th moment of the number of guesses for every ρ > 0. Upper and lower bounds on the ρ-th moment of ranking functions, expressed in terms of the Rényi entropies, were derived by Arikan [1], Boztaş [10], followed by recent improvements in the non-asymptotic regime by Sason and Verdú [68]. Although if |X | is small, it is straightforward to evaluate numerically the guessing moments, the benefit of bounds expressed in terms of Rényi entropies is particularly relevant when dealing with a random vector X k = (X 1 , .…”

“…Sason and Verdú recently derived in [68] improved non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints, and non-asymptotic bounds on the reliability function of a DMS, tightening the bounds in [23].…”

“…The motivation for this work is rooted in the diverse information-theoretic applications of Rényi measures [62]. These include (but are not limited to) asymptotically tight bounds on guessing moments [1], informationtheoretic applications such as guessing subject to distortion [2], joint source-channel coding and guessing with application to sequential decoding [3], guessing with a prior access to a malicious oracle [14], guessing while allowing the guesser to give up and declare an error [50], guessing in secrecy problems [56], [75], guessing with limited memory [64], and guessing under source uncertainty [74]; encoding tasks [12], [13]; Bayesian hypothesis testing [8], [67], [79], and composite hypothesis testing [71], [77]; Rényi generalizations of the rejection sampling problem in [35], motivated by the communication complexity in distributed channel simulation, where these generalizations distinguish between causal and non-causal sampler scenarios [52]; Wyner's common information in distributed source simulation under Rényi divergence measures [87]; various other source coding theorems [15], [23], [24], [36], [49], [50], [68], [76], [78], [79], channel coding theorems [4], [5], [26], [60], [66], [78], [79], [86], including coding theorems in quantum information theory…”

“…The term scaled cumulant generating function is used in view of[68, Remark 20].December 11, 2018 DRAFT…”