Information Theoretic Proofs of Entropy Power Inequalities

Abstract

While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verd\'u used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.Comment: submitted for publication in the IEEE Transactions on Information Theory, revised versio

A New Tight Upper Bound on the Entropy of Sums

“…Under this finite logarithmic constraint, the differential entropy of X is well defined and is such that −∞ ≤ h(X) < +∞ (Proposition 1, [5]) and that of Y exists and is finite (Lemma 1, [5]). Also, when X ∈ L, the identity I(X + Z; Z) = h(X + Z) − h(X) always holds (Lemma 1, [5]).…”

“…The result of Theorem 1 is more powerful that the IIE in Equation (5). Indeed, using the fact that h(Z) ≤ h(X + Z), inequality Equation (10) gives the looser inequality:…”

“…Rioul proved that the de Bruijn's identity holds at = 0 + for any finite-variance RV Z (Proposition 7, p. 39, [5]). …”

“…In the case of a Gaussian-distributed U, the family {U t } t≥0 has the same distribution as √ tU t , and it is already known that the entropy (and actually even the entropy power) of Y is concave in t ((Section VII, p. 51, [5]) and [19]). …”

“…by virtue of the fact that h(Y l ) is concave in ≥ 0 (Section VII, p. 51, [5]). Using the FII Equation (3) on Y l we obtain…”