MMSE Bounds for Additive Noise Channels Under Kullback–Leibler Divergence Constraints on the Input Distribution

Dytso, Alex; Fauß, Michael; Zoubir, Abdelhak M.; Poor, H. Vincent

doi:10.1109/tsp.2019.2951221

Cited by 18 publications

(28 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where Example 11.4]. This expression together with the integral version of Jaffer's identity in (36) lead to the following simple relationship between the conditional expectation and the conditional cumulants.…”

Section: A the Univariate Casementioning

confidence: 99%

See 1 more Smart Citation

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

Dytso

Poor

Shitz

2020

2020 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

Consider a channel Y = X + N where X is an n-dimensional random vector, and N is a multivariate Gaussian vector with a full-rank covariance matrix KN. The object under consideration in this paper is the conditional mean of X given Y = y, that is y → E[X|Y = y]. Several identities in the literature connect E[X|Y = y] to other quantities such as the conditional variance, score functions, and higher-order conditional moments. The objective of this paper is to provide a unifying view of these identities.In the first part of the paper, a general derivative identity for the conditional mean estimator is derived. Specifically, for the Markov chain U ↔ X ↔ Y, it is shown that the Jacobian matrix of E[U|Y = y] is given by K −1 N Cov(X, U|Y = y) where Cov(X, U|Y = y) is the conditional covariance.In the second part of the paper, via various choices of the random vector U, the new identity is used to recover and generalize many of the known identities and derive some new identities. First, a simple proof of the Hatsel and Nolte identity for the conditional variance is shown. Second, a simple proof of the recursive identity due to Jaffer is provided. The Jaffer identity is then further explored, and several equivalent stamens are derived, such as an identity for the higher-order conditional expectation (i.e., E[X k |Y]) in terms of the derivatives of the conditional expectation. Third, a new fundamental connection between the conditional cumulants and the conditional expectation is demonstrated. In particular, in the univariate case, it is shown that the k-th derivative of the conditional expectation is proportional to the (k +1)-th conditional cumulant. A similar expression is derived in the multivariate case.The third part of the paper considers various applications of the derived identities (mostly in the scalar case). In a first application, using the new identity for higher-order derivatives of the conditional expectation, a power series representation of the conditional expectation is derived. The power series representation, together with the Lagrange inversion theorem, is then used to find an expression for the compositional inverse of y → E[X|Y = y]. In a second application, the conditional expectation is viewed as a random variable and the probability distribution of E[X|Y ] and probability distribution of the estimator error (X − E[X|Y ]) are derived. In the third application, the new identities are used to show that the higher-order conditional expectations and the conditional cumulants depended on the joint distribution only through the marginal of Y . This observation is then used to construct consistent estimators (known as the empirical Bayes estimators) of the higher-order conditional expectations and the conditional cumulants from an independent and identically distributed sequence Y1, . . . , Yn.

show abstract

Section: A the Univariate Casementioning

confidence: 99%

“…5. There exists a large collection of lower bounds on estimation error for continuous distributions [34]- [36] of which the Cramér-Rao bound is arguably the most popular. However, lower bounds on the MMSE of other distributions are rare.…”

Section: Representations Of the Mmsementioning

confidence: 99%

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

Dytso

Poor

Shitz

2020

2020 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

show abstract

“…This paper extends the bounds in [10] and [11] to a weighted sum of MMSEs. Similar to [10] and [11], the bounds derived in this paper are obtained by constraining the input distribution to be ε-close to a Gaussian reference distribution in terms of the KL divergence. The estimators that attain the upper bound are minimax robust against deviations from the assumed input distribution.…”

Section: Introductionmentioning

confidence: 75%

“…The practical implications of this phenomenon is that, our bounds hold for a larger set of distributions. This property of the proposed bounds has been discussed in detail in [11]. An example of such a prior distribution is a uniform distribution over a Kball.…”

Section: B Generalized Gaussian and Uniform Distributionsmentioning

confidence: 93%

“…The minimum MSE (MMSE) plays a central role in statistics [1], [2], information theory [3], [4], signal processing [5]- [7], and has close connections to entropy and mutual information [8], [9]. In [10] and [11], lower and upper bounds on the MMSE are derived when the random variable of interest is contaminated by additive Gaussian noise and its distribution is ε-close to a Gaussian reference distribution in terms of the Kullback-Leibler (KL) divergence. The estimator that attains this upper bound is shown to be minimax robust in the sense that it minimizes the maximum MMSE over the set of feasible distributions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tight Bounds on the Weighted Sum of MMSEs with Applications in Distributed Estimation

Faub

Zoubir

Dytso

et al. 2019

2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)

Self Cite

View full text Add to dashboard Cite

In this paper, tight upper and lower bounds are derived on the weighted sum of minimum mean-squared errors for additive Gaussian noise channels. The bounds are obtained by constraining the input distribution to be close to a Gaussian reference distribution in terms of the Kullback-Leibler divergence. The distributions that attain these bounds are shown to be Gaussian whose covariance matrices are defined implicitly via systems of matrix equations. Furthermore, the estimators that attain the upper bound are shown to be minimax robust against deviations from the assumed input distribution. The lower bound provides a potentially tighter alternative to well-known inequalities such as the Cramér-Rao lower bound. Numerical examples are provided to verify the theoretical findings of the paper. The results derived in this paper can be used to obtain performance bounds, robustness guarantees, and engineering guidelines for the design of local estimators for distributed estimation problems which commonly arise in wireless communication systems and sensor networks.

show abstract

A variational interpretation of the Cramér–Rao bound

2021

View full text Add to dashboard Cite

MMSE Bounds for Additive Noise Channels Under Kullback–Leibler Divergence Constraints on the Input Distribution

Cited by 18 publications

References 29 publications

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

Tight Bounds on the Weighted Sum of MMSEs with Applications in Distributed Estimation

A variational interpretation of the Cramér–Rao bound

Contact Info

Product

Resources

About