Tight Bounds on the Rényi Entropy via Majorization with Applications to Guessing and Compression

Sason, Igal

doi:10.3390/e20120896

Cited by 32 publications

(40 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To prove our results, we use ideas and techniques from majorization theory [39], a mathematical framework that has been proved to be very much useful in information theory (e.g., see [5], [6], [7], [8], [17], [23], [24], [48] and references therein). In this section we recall the notions and results that are relevant to our context.…”

Section: Preliminary Resultsmentioning

confidence: 99%

Minimum-Entropy Couplings and Their Applications

Cicalese

Gargano

Vaccaro

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Given two discrete random variables X and Y, with probability distributions p = (p1, . . . , pn) and q = (q1, . . . , qm), respectively, denote by C(p, q) the set of all couplings of p and q, that is, the set of all bivariate probability distributions that have p and q as marginals. In this paper, we study the problem of finding a joint probability distribution in C(p, q) of minimum entropy (equivalently, a coupling that maximizes the mutual information between X and Y ), and we discuss several situations where the need for this kind of optimization naturally arises. Since the optimization problem is known to be NP-hard, we give an efficient algorithm to find a joint probability distribution in C(p, q) with entropy exceeding the minimum possible at most by 1 bit, thus providing an approximation algorithm with an additive gap of at most 1 bit. Leveraging on this algorithm, we extend our result to the problem of finding a minimum-entropy joint distribution of arbitrary k ≥ 2 discrete random variables X1, . . . , X k , consistent with the known k marginal distributions of the individual random variables X1, . . . , X k . In this case, our algorithm has an additive gap of at most log k from optimum. We also discuss several related applications of our findings and extensions of our results to entropies different from the Shannon entropy.

show abstract

Section: Preliminary Resultsmentioning

confidence: 99%

Minimum-Entropy Couplings and Their Applications

Cicalese

Gargano

Vaccaro

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…for all t > 0 (although, unlike the asymptotic result in (149), the refined bound for a finite n does not lend itself to a closed-form expression as a function of n; see also [63,Remark 3], which provides such a refinement of the bound on D(Q U n ) for finite n in a different approach). From (141), (146) and (150), it follows similarly to (153) that for all ρ > 1…”

Section: Illustration Of Theorem 7 and Further Resultsmentioning

confidence: 99%

“…the asymptotic result in (149) can be obtained from [63, Lemma 4] and vice versa; however, in [63], the focus is on the Rényi divergence from the equiprobable distribution, whereas the result in (149) is obtained by specializing the asymptotic expression in (134) for a general f -divergence. Note also that the result in [63,Lemma 4] is restricted to α > 0, whereas the result in (149) and (150) covers all values of α ∈ R. 26 In view of (146), (149), (153), (155), and the special cases of the Alpha divergences in (140)-(144), it follows that for all ρ > 1 and for all integer n ≥ 2 max Q∈Pn(ρ)…”

Section: Illustration Of Theorem 7 and Further Resultsmentioning

confidence: 99%

On Data-Processing and Majorization Inequalities for f-Divergences with Applications

Sason

2019

Entropy

Self Cite

View full text Add to dashboard Cite

This paper is focused on derivations of data-processing and majorization inequalities for f -divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified way, and new bounds are obtained and exemplified numerically. Another application is related to a study of the quality of approximating a probability mass function, induced by the leaves of a Tunstall tree, by an equiprobable distribution. The compression rates of finite-length Tunstall codes are further analyzed for asserting their closeness to the Shannon entropy of a memoryless and stationary discrete source. Almost all the analysis is relegated to the appendices, which form the major part of this manuscript.

show abstract

“…The large deviation behavior of guessing was studied in [13,14]. The relation between guessing and variable-length lossless source coding was explored in [3,15,16].…”

Section: Related Workmentioning

confidence: 99%

Guessing with Distributed Encoders

Bracher

Lapidoth

Pfister

2019

Entropy

View full text Add to dashboard Cite

Two correlated sources emit a pair of sequences, each of which is observed by a different encoder. Each encoder produces a rate-limited description of the sequence it observes, and the two descriptions are presented to a guessing device that repeatedly produces sequence pairs until correct. The number of guesses until correct is random, and it is required that it have a moment (of some prespecified order) that tends to one as the length of the sequences tends to infinity. The description rate pairs that allow this are characterized in terms of the Rényi entropy and the Arimoto–Rényi conditional entropy of the joint law of the sources. This solves the guessing analog of the Slepian–Wolf distributed source-coding problem. The achievability is based on random binning, which is analyzed using a technique by Rosenthal.

show abstract

Tight Bounds on the Rényi Entropy via Majorization with Applications to Guessing and Compression

Cited by 32 publications

References 91 publications

Minimum-Entropy Couplings and Their Applications

Minimum-Entropy Couplings and Their Applications

On Data-Processing and Majorization Inequalities for f-Divergences with Applications

Guessing with Distributed Encoders

Contact Info

Product

Resources

About