Mutual Information, Fisher Information, and Efficient Coding

Abstract

Fisher information is generally believed to represent a lower bound on mutual information (Brunel & Nadal, 1998), a result that is frequently used in the assessment of neural coding efficiency. However, we demonstrate that the relation between these two quantities is more nuanced than previously thought. For example, we find that in the small noise regime, Fisher information actually provides an upper bound on mutual information. Generally our results show that it is more appropriate to consider Fisher information as an approximation rather than a bound on mutual information. We analytically derive the correspondence between the two quantities and the conditions under which the approximation is good. Our results have implications for neural coding theories and the link between neural population coding and psychophysically measurable behavior. Specifically, they allow us to formulate the efficient coding problem of maximizing mutual information between a stimulus variable and the response of a neural population in terms of Fisher information. We derive a signature of efficient coding expressed as the correspondence between the population Fisher information and the distribution of the stimulus variable. The signature is more general than previously proposed solutions that rely on specific assumptions about the neural tuning characteristics. We demonstrate that it can explain measured tuning characteristics of cortical neural populations that do not agree with previous models of efficient coding.

Closed-Loop Estimation of Retinal Network Sensitivity by Local Empirical Linearization

“…On one hand, in the small frequency regime, sensitivity increases with frequency, i.e., decreases with stimulus power. This result is classic: when the input noise is small compared to stimulus, the best coding strategy for maximizing information is to whiten the input signal to obtain a flat output spectrum, which is obtained by having the squared sensitivity be inversely proportional to the stimulus power (Rieke et al, 1996; Wei and Stocker, 2016). On the other hand, at high frequencies, the input noise is too high (relative to the stimulus power) for the stimulus to be recovered.…”

Reliability as Lindley Information

“…In particular, for scenarios with monotonic relationships between the observed and latent variables, and smooth measurement error, as the error approaches zero and at the same time becomes more normal in shape, the inequality becomes an equality (Wei & Stocker, 2016). Wei and Stocker (2016) give a proof of this for the unidimensional case that can be applied straightforwardly to the multidimensional case. Assume a generalized version of Equation 7,…”

“…Depending on the scenario, the inequality in Equation 26 can be reversed in direction to become an upper bound, or can become an equality. In particular, for scenarios with monotonic relationships between the observed and latent variables, and smooth measurement error, as the error approaches zero and at the same time becomes more normal in shape, the inequality becomes an equality (Wei & Stocker, 2016).…”

“…Wei and Stocker (2016) note that, because f is invertible, the Lindley information between X and θ, ɩ(X, θ), is equal to ɩ(X, ϑ). Given this, and that the entropy H(X|ϑ) can be reexpressed as H(ε), ɩ(X, θ) can be expressed…”

“…Using standard formulas for the entropy and Fisher information under a transformation, Wei and Stocker (2016) note that…”