Optimal Encoding in Stochastic Latent-Variable Models

Abstract

In this work we explore encoding strategies learned by statistical models of sensory coding in noisy spiking networks. Early stages of sensory communication in neural systems can be viewed as encoding channels in the information-theoretic sense. However, neural populations face constraints not commonly considered in communications theory. Using restricted Boltzmann machines as a model of sensory encoding, we find that networks with sufficient capacity learn to balance precision and noise-robustness in order to adaptively communicate stimuli with varying information content. Mirroring variability suppression observed in sensory systems, informative stimuli are encoded with high precision, at the cost of more variable responses to frequent, hence less informative stimuli. Curiously, we also find that statistical criticality in the neural population code emerges at model sizes where the input statistics are well captured. These phenomena have well-defined thermodynamic interpretations, and we discuss their connection to prevailing theories of coding and statistical criticality in neural populations.

The Information Theory of Developmental Pruning: Optimizing Global Network Architecture Using Local Synaptic Rules

“…Presumably, neurons in the brain use a local heuristic that can be computed based on information available to single synapses. It was recently shown that local activity statistics indeed correlate with importance as assessed by FI (Rule et al, 2020). In the case of the RBM this correspondence is exact; an entry of the FIM for an RBM has the form (Rule et al, 2020):…”

“…Through sampling we can average over ,ℎ . For instance, for two weights and connecting a presynaptic to a postsynaptic neuron, respectively, we get (Rule et al, 2020):…”

“…We started by tting overcomplete RBMs to circular, binarized patches sliced out of images of natural scenes (see Figure 1 A). Parameter-wise FI was computed by our activity-dependent estimate of FI from Equation 2 (see Appendix A.1 and (Rule et al, 2020)). For these relatively small models, we computed the full FIM, which typically turned out to be sparse (see Figure 1 B).…”

“…By adjusting their hidden representations as to maximize the likelihood of the data (Hinton, 2002), RBMs learn in an unsupervised fashion that resembles the Hebbian learning seen in neural networks (Whittington and Bogacz, 2017;Kermiche, 2019). RBMs have been used as statistical models of sensory coding (Zanotto et al, 2017;Gardella et al, 2018;Rule et al, 2020). Furthermore, multiple RBMs can be stacked to build a Deep Boltzmann Machine (DBM) (Hinton and Salakhutdinov, 2006) to model the hierarchical, bidirectional computation of the visual system (Hochstein and Ahissar, 2002;Turcsany et al, 2014).…”

“…In contrast to identifying parameters worth keeping, we aim to identify parameters worth removing. For RBMs, the diagonal of the FIM can be estimated locally based on the ring rates of units and their coincidence (Rule et al, 2020;Deistler et al, 2018). In principle, this estimate of parameter importance is available to individual synapses and neurons, and could thus guide the search for e cient representations during neurodevelopmental pruning.…”