On Nonparametric Estimation of the Fisher Information

Cao, Wei; Dytso, Alex; Fauß, Michael; Poor, H. Vincent; Feng, Gang

doi:10.1109/isit44484.2020.9174450

Cited by 4 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We take the kernel to be k(y) = φ(y) = 1 √ 2π e − y 2 2 ; and 2) Estimate f (k) Y (y) by taking the derivative of (209) k times. The above estimators are inspired by the estimators of the score function and the Fisher information studied in [44], [45] and [46].…”

Section: A Empirical Bayes For Higher-order Conditional Momentsmentioning

confidence: 99%

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

Dytso

Poor

Shitz

2020

2020 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

Consider a channel Y = X + N where X is an n-dimensional random vector, and N is a multivariate Gaussian vector with a full-rank covariance matrix KN. The object under consideration in this paper is the conditional mean of X given Y = y, that is y → E[X|Y = y]. Several identities in the literature connect E[X|Y = y] to other quantities such as the conditional variance, score functions, and higher-order conditional moments. The objective of this paper is to provide a unifying view of these identities.In the first part of the paper, a general derivative identity for the conditional mean estimator is derived. Specifically, for the Markov chain U ↔ X ↔ Y, it is shown that the Jacobian matrix of E[U|Y = y] is given by K −1 N Cov(X, U|Y = y) where Cov(X, U|Y = y) is the conditional covariance.In the second part of the paper, via various choices of the random vector U, the new identity is used to recover and generalize many of the known identities and derive some new identities. First, a simple proof of the Hatsel and Nolte identity for the conditional variance is shown. Second, a simple proof of the recursive identity due to Jaffer is provided. The Jaffer identity is then further explored, and several equivalent stamens are derived, such as an identity for the higher-order conditional expectation (i.e., E[X k |Y]) in terms of the derivatives of the conditional expectation. Third, a new fundamental connection between the conditional cumulants and the conditional expectation is demonstrated. In particular, in the univariate case, it is shown that the k-th derivative of the conditional expectation is proportional to the (k +1)-th conditional cumulant. A similar expression is derived in the multivariate case.The third part of the paper considers various applications of the derived identities (mostly in the scalar case). In a first application, using the new identity for higher-order derivatives of the conditional expectation, a power series representation of the conditional expectation is derived. The power series representation, together with the Lagrange inversion theorem, is then used to find an expression for the compositional inverse of y → E[X|Y = y]. In a second application, the conditional expectation is viewed as a random variable and the probability distribution of E[X|Y ] and probability distribution of the estimator error (X − E[X|Y ]) are derived. In the third application, the new identities are used to show that the higher-order conditional expectations and the conditional cumulants depended on the joint distribution only through the marginal of Y . This observation is then used to construct consistent estimators (known as the empirical Bayes estimators) of the higher-order conditional expectations and the conditional cumulants from an independent and identically distributed sequence Y1, . . . , Yn.

show abstract

Section: A Empirical Bayes For Higher-order Conditional Momentsmentioning

confidence: 99%

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

Dytso

Poor

Shitz

2020

2020 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, we demonstrate the difference in the bounds on the convergence rates between Bhattacharya’s estimator and its clipped version by comparing the sample complexity of the two estimators, that is, the required number of samples to guarantee a given accuracy with a given confidence. MATLAB implementations of both estimators, as well as the code used to generate the figures below, can be found in [ 22 ].…”

Section: Estimation Of the Fisher Information Of A Random Variable In Gaussian Noisementioning

confidence: 99%

“…This trivially implies that X is

-sub-Gaussian with

. In order to make the comparison as fair as possible, the parameters of the kernel estimators,

, and

, are not chosen according to Theorem 4 or Theorem 5, but are calculated by numerically minimizing the required number of samples; see [ 22 ] for details.…”

Section: Estimation Of the Fisher Information Of A Random Variable In Gaussian Noisementioning

confidence: 99%

“…Finally, the problem of obtaining estimator-independent bounds on the sample complexity of Fisher information falls under the umbrella of estimation of nonlinear functionals; see, for example, [ 12 ]. Most of the commonly used information measures, such as entropy, relative entropy, and mutual information, are nonlinear functionals, and their estimation has recently received considerable attention; the interested reader is referred to [ 13 , 14 , 15 , 16 , 17 ] and the references therein.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Finite-Sample Bounds on the Accuracy of Plug-In Estimators of Fisher Information

Cao

Dytso

Fauß

et al. 2021

Entropy

Self Cite

View full text Add to dashboard Cite

Finite-sample bounds on the accuracy of Bhattacharya’s plug-in estimator for Fisher information are derived. These bounds are further improved by introducing a clipping step that allows for better control over the score function. This leads to superior upper bounds on the rates of convergence, albeit under slightly different regularity conditions. The performance bounds on both estimators are evaluated for the practically relevant case of a random variable contaminated by Gaussian noise. Moreover, using Brown’s identity, two corresponding estimators of the minimum mean-square error are proposed.

show abstract

“…Estimators for these quantities can be found e.g. in [3,4,8,20]. The proposition above can be generalized to richer classes of functions K, e.g.…”

Section: Compensating For Gainmentioning

confidence: 99%

A Unification of Weighted and Unweighted Particle Filters

Abedi,

Surace,

Pfister

2020

Preprint

View full text Add to dashboard Cite

Particle filters (PFs), which are successful methods for approximating the solution of the filtering problem, can be divided into two types: weighted and unweighted PFs. It is wellknown that weighted PFs suffer from the weight degeneracy and curse of dimensionality. To sidestep these issues, unweighted PFs have been gaining attention, though they have their own challenges. The existing literature on these types of PFs is based on distinct approaches. In order to establish a connection, we put forward a framework that unifies weighted and unweighted PFs in the continuoustime filtering problem. We show that the stochastic dynamics of a particle system described by a pair process, representing particles and their importance weights, should satisfy two necessary conditions in order for its distribution to match the solution of the Kushner-Stratonovich equation.In particular, we demonstrate that the bootstrap particle filter (BPF), which relies on importance sampling, and the feedback particle filter (FPF), which is an unweighted PF based on optimal control, arise as special cases from a broad class, and that there is a smooth transition between the two. The freedom in designing the PF dynamics opens up potential ways to address the existing issues in the aforementioned algorithms, namely weight degeneracy in the BPF and gain estimation in the FPF.

show abstract

On Nonparametric Estimation of the Fisher Information

Cited by 4 publications

References 20 publications

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

A General Derivative Identity for the Conditional Mean Estimator in Gaussian Noise and Some Applications

Finite-Sample Bounds on the Accuracy of Plug-In Estimators of Fisher Information

A Unification of Weighted and Unweighted Particle Filters

Contact Info

Product

Resources

About