Fisher Information Properties

Zegers, Pablo

doi:10.3390/e17074918

Cited by 45 publications

(36 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar inequality is derived in [24] for the mutual Fisher information. The mutual Fisher information is rather similar to V[E[u θ |e]] than G I θ .…”

Section: Monotonicity Of Asymptotic Efficiencymentioning

confidence: 71%

See 1 more Smart Citation

Efficiency Bound of Local Z-Estimators on Discrete Sample Spaces

Kanamori

2016

Entropy

View full text Add to dashboard Cite

Many statistical models over a discrete sample space often face the computational difficulty of the normalization constant. Because of that, the maximum likelihood estimator does not work. In order to circumvent the computation difficulty, alternative estimators such as pseudo-likelihood and composite likelihood that require only a local computation over the sample space have been proposed. In this paper, we present a theoretical analysis of such localized estimators. The asymptotic variance of localized estimators depends on the neighborhood system on the sample space. We investigate the relation between the neighborhood system and estimation accuracy of localized estimators. Moreover, we derive the efficiency bound. The theoretical results are applied to investigate the statistical properties of existing estimators and some extended ones.

show abstract

“…A similar inequality is derived in [24] for the mutual Fisher information. The mutual Fisher information is rather similar to V[E[u θ |e]] than G I θ .…”

Section: Monotonicity Of Asymptotic Efficiencymentioning

confidence: 71%

“…The mutual Fisher information is rather similar to V[E[u θ |e]] than G I θ . Theorem 13 of [24] corresponds to the one-dimensional version of the inequality…”

Section: Monotonicity Of Asymptotic Efficiencymentioning

confidence: 99%

Efficiency Bound of Local Z-Estimators on Discrete Sample Spaces

Kanamori

2016

Entropy

View full text Add to dashboard Cite

show abstract

“…So by replacing W and U A in (45), or equivalently P out (y|z) = (1 − )δ(y − π(z)) + δ(y + π(z)) into (44), one obtains the Shannon capacity of the BSC channel R ∞ pot = 1 − h 2 ( ) where h 2 is the binary entropy function. Using (43) this map also gives the algorithmic threshold…”

Section: B Binary Symmetric Channelmentioning

confidence: 99%

Universal Sparse Superposition Codes With Spatial Coupling and GAMP Decoding

Barbier

Dia

Macris

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Sparse superposition codes, or sparse regression codes, constitute a new class of codes which was first introduced for communication over the additive white Gaussian noise (AWGN) channel. It has been shown that such codes are capacity-achieving over the AWGN channel under optimal maximum-likelihood decoding as well as under various efficient iterative decoding schemes equipped with power allocation or spatially coupled constructions. Here, we generalize the analysis of these codes to a much broader setting that includes all memoryless channels. We show, for a large class of memoryless channels, that spatial coupling allows an efficient decoder, based on the generalized approximate message-passing (GAMP) algorithm, to reach the potential (or Bayes optimal) threshold of the underlying (or uncoupled) code ensemble. Moreover, we argue that spatially coupled sparse superposition codes universally achieve capacity under GAMP decoding by showing, through analytical computations, that the error floor vanishes and the potential threshold tends to capacity as one of the code parameter goes to infinity. Furthermore, we provide a closed form formula for the algorithmic threshold of the underlying code ensemble in terms of a Fisher information. Relating an algorithmic threshold to a Fisher information has theoretical as well as practical importance. Our proof relies on the state evolution analysis and uses the potential method developed in the theory of low-density parity-check (LDPC) codes and compressed sensing.

show abstract

“…It is important to note that the term d J(I) ( f {S}n ; f {L}n;ψ,θ ) θ k corresponds to the relative information [21]. This term can be re-expressed as:…”

Section: The Kullback-leibler Divergence Is Usefulmentioning

confidence: 99%

Information Theoretical Measures for Achieving Robust Learning Machines

Zegers

Frieden

Alarcón³

et al. 2016

Entropy

Self Cite

View full text Add to dashboard Cite

Information theoretical measures are used to design, from first principles, an objective function that can drive a learning machine process to a solution that is robust to perturbations in parameters. Full analytic derivations are given and tested with computational examples showing that indeed the procedure is successful. The final solution, implemented by a robust learning machine, expresses a balance between Shannon differential entropy and Fisher information. This is also surprising in being an analytical relation, given the purely numerical operations of the learning machine.

show abstract

Fisher Information Properties

Cited by 45 publications

References 45 publications

Efficiency Bound of Local Z-Estimators on Discrete Sample Spaces

Efficiency Bound of Local Z-Estimators on Discrete Sample Spaces

Universal Sparse Superposition Codes With Spatial Coupling and GAMP Decoding

Information Theoretical Measures for Achieving Robust Learning Machines

Contact Info

Product

Resources

About