Modelling Retinal Feature Detection With Deep Belief Networks In A Simulated Environment

Turcsány, Diána; Bargieła, Andrzej; Maul, Tomás

doi:10.7148/2014-0364

Cited by 7 publications

(5 citation statements)

References 27 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RBMs have been used as statistical models of sensory coding [30,[35][36][37]. Furthermore, multiple RBMs can be stacked to build a deep Boltzmann machine (DBM) [38] to model the hierarchical, bidirectional computation of the visual system [39,40]. Such models have been used, e.g., to explore consolidation processes in declarative memory [41] or how loss of input could lead to visual hallucinations in Charles Bonnet Syndrome [42].…”

Section: Introductionmentioning

confidence: 99%

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

2021

View full text Add to dashboard Cite

During development, biological neural networks produce more synapses and neurons than needed. Many of these synapses and neurons are later removed in a process known as neural pruning. Why networks should initially be over-populated, and the processes that determine which synapses and neurons are ultimately pruned, remains unclear. We study the mechanisms and significance of neural pruning in model neural networks. In a deep Boltzmann machine model of sensory encoding, we find that (1) synaptic pruning is necessary to learn efficient network architectures that retain computationally-relevant connections, (2) pruning by synaptic weight alone does not optimize network size and (3) pruning based on a locally-available measure of importance based on Fisher information allows the network to identify structurally important vs. unimportant connections and neurons. This locally-available measure of importance has a biological interpretation in terms of the correlations between presynaptic and postsynaptic neurons, and implies an efficient activity-driven pruning rule. Overall, we show how local activity-dependent synaptic pruning can solve the global problem of optimizing a network architecture. We relate these findings to biology as follows: (I) Synaptic over-production is necessary for activity-dependent connectivity optimization. (II) In networks that have more neurons than needed, cells compete for activity, and only the most important and selective neurons are retained. (III) Cells may also be pruned due to a loss of synapses on their axons. This occurs when the information they convey is not relevant to the target population.

show abstract

Section: Introductionmentioning

confidence: 99%

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

2021

View full text Add to dashboard Cite

show abstract

“…Although RBMs can be made to resemble spiking systems by extending their dynamics in time [ 15 ], statistical models of spiking populations commonly consider the synchronous case [ 16 ], which models only zero-lag dependencies between spikes. RBMs have been used as a statistical model to study retinal population coding [ 17 , 18 ], and multi-layer RBMs have been used as a model of retinal computation [ 19 ]. RBMs can be connected to more biologically plausible spiking models [ 20 ], but added biological plausibility does not not change essential statistical features that we wish to study, and obscures relevant thermodynamic interpretations.…”

Section: Introductionmentioning

confidence: 99%

Optimal Encoding in Stochastic Latent-Variable Models

Rule

Sorbaro

Hennig

2020

Entropy

View full text Add to dashboard Cite

In this work we explore encoding strategies learned by statistical models of sensory coding in noisy spiking networks. Early stages of sensory communication in neural systems can be viewed as encoding channels in the information-theoretic sense. However, neural populations face constraints not commonly considered in communications theory. Using restricted Boltzmann machines as a model of sensory encoding, we find that networks with sufficient capacity learn to balance precision and noise-robustness in order to adaptively communicate stimuli with varying information content. Mirroring variability suppression observed in sensory systems, informative stimuli are encoded with high precision, at the cost of more variable responses to frequent, hence less informative stimuli. Curiously, we also find that statistical criticality in the neural population code emerges at model sizes where the input statistics are well captured. These phenomena have well-defined thermodynamic interpretations, and we discuss their connection to prevailing theories of coding and statistical criticality in neural populations.

show abstract

“…RBMs have been used as statistical models of sensory coding (Zanotto et al, 2017;Gardella et al, 2018;Rule et al, 2020). Furthermore, multiple RBMs can be stacked to build a Deep Boltzmann Machine (DBM) (Hinton and Salakhutdinov, 2006) to model the hierarchical, bidirectional computation of the visual system (Hochstein and Ahissar, 2002;Turcsany et al, 2014). Such DBM models have been used, e.g., to explore how loss of input could lead to visual hallucinations in Charles Bonnet Syndrome (Reichert et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

The Information Theory of Developmental Pruning: Optimizing Global Network Architecture Using Local Synaptic Rules

Scholl¹,

Rule

Hennig

2020

Preprint

View full text Add to dashboard Cite

During development, biological neural networks produce more synapses and neurons than needed. Many of these synapses and neurons are later removed in a process known as neural pruning. Why networks should initially be over-populated, and processes that determine which synapses and neurons are ultimately pruned, remains unclear. We study the mechanisms and significance of neural pruning in model neural network. In a deep Boltzmann machine model of sensory encoding, we find that (1) synaptic pruning is necessary to learn efficient network architectures that retain computationally-relevant connections, (2) pruning by synaptic weight alone does not optimize network size and (3) pruning based on a locally-available proxy for "sloppiness" based on Fisher Information allows the network to identify structurally important vs. unimportant connections and neurons. This locally-available measure of importance has a biological interpretation in terms of the correlations between presynaptic and postsynaptic neurons, and implies an efficient activity-driven pruning rule. Overall, we show how local activity-dependent synaptic pruning can solve the global problem of optimizing a network architecture. We relate these findings to biology as follows: (I) Synaptic over-production is necessary for activity-dependent connectivity optimization. (II) In networks that have more neurons than needed, cells compete for activity, and only the most important and selective neurons are retained. (III) Cells may also be pruned due to a loss of synapses on their axons. This occurs when the information they convey is not relevant to the target population.

show abstract

Modelling Retinal Feature Detection With Deep Belief Networks In A Simulated Environment

Cited by 7 publications

References 27 publications

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

Optimal Encoding in Stochastic Latent-Variable Models

The Information Theory of Developmental Pruning: Optimizing Global Network Architecture Using Local Synaptic Rules

Contact Info

Product

Resources

About