Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation

Scellier, Benjamin; Bengio, Yoshua

doi:10.48550/arxiv.1602.05179

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Equation (8) and Equation (9). The B conv,l rand and B f c,l rand were random generated matrixs for each hidden layer l before network learning and would not be further updated during learning.…”

Section: E the Local Synaptic Weight Consolidationmentioning

confidence: 99%

“…Some researchers tried to find these answers by taking efforts to the inner-side research of the DNN itself, such as constructing specific network structures, designing more powerful cost functions, or building finer visualization tools so as to open the black box of DNN. These efforts were efficient and also contributed to further development of DNNs [8], [9], [10], however, in this paper, we think that there might be still an alternative and easier approach to achieve these goals especially on robust and efficient computation, by turning into the biological neural networks and then getting efficientcomputation inspirations from them towards the human-level robust computation [11], [12], [13], [14], [15], [16], [17], [18].…”

mentioning

confidence: 99%

See 1 more Smart Citation

Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation

Zhang

Jia

Cheng

et al. 2020

Preprint

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) contain more biology-realistic structures and biology-inspired learning principles compared with that in standard Artificial Neural Networks (ANNs). SNNs were considered as the third generation of ANNs, powerful on robust computation with a low computational cost. The dynamic neurons in SNNs are non-differential, containing decayed historical states and generating event-based spikes after their states reaching the firing threshold. This dynamic characteristic of SNNs made it hard to be directly trained with standard back propagation (BP) which is considered not biologically plausible. In this paper, a Biologically-plausible Reward Propagation (BRP) algorithm is proposed and applied on a SNN architecture with both spiking-convolution (with both 1D and 2D convolutional kernels) and full-connection layers. Different with standard BP that propagated the error signals from post to pre synaptic neurons layer by layer, the BRP propagated the target labels instead of target errors directly from the output layer to all of the pre hidden layers. This effort was more consistent with the top-down reward-guiding learning in cortical columns of the neocortex. Then synaptic modifications with only local gradient differences were induced with pseudo-BP that might also be replaced with Spike-Timing Dependent Plasticity (STDP). The performance of the proposed BRP-SNN was further verified on spatial (including MNIST and Cifar-10) and temporal (including TIDigits and DvsGesture) tasks. The experimental result showed that the BRP played roles on convergent learning of SNN, reached higher accuracy compared with other state-of-the-art SNN algorithms, and saved more than 50% computational cost compared with that on ANNs. We think the introduction of biologically-plausible learning rules to the training procedure of biologically-realistic SNNs might give us more hints and inspirations towards a better understanding of the intelligent nature of the biological system.

show abstract

Section: E the Local Synaptic Weight Consolidationmentioning

confidence: 99%

mentioning

confidence: 99%

Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation

Zhang

Jia

Cheng

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Both supervised and unsupervised learning depend on the backpropagation algorithm, which is traditionally considered biologically implausible (Glaser, Benjamin, Farhoodi & Kording, 2018). Recent theoretical work suggests that deep learning with an algorithm similar to backpropagation might be biologically feasible (Lillicrap et al, 2016;Guerguiev, Lillicrap, and Richards, 2017;Kording & K önig 2001;Scellier & Bengio, 2016;Hinton and McClelland 1988). However, supervised training on millions of labelled stimuli is still an ecologically unrealistic requirement.…”

Section: Building Network That Learn In More Ecologically and Biologi...mentioning

confidence: 99%

Deep Learning for Cognitive Neuroscience

Storrs¹,

Kriegeskorte²

2019

Preprint

View full text Add to dashboard Cite

Neural network models can now recognise images, understand text, translate languages, and play many human games at human or superhuman levels. These systems are highly abstracted, but are inspired by biological brains and use only biologically plausible computations. In the coming years, neural networks are likely to become less reliant on learning from massive labelled datasets, and more robust and generalisable in their task performance. From their successes and failures, we can learn about the computational requirements of the different tasks at which brains excel. Deep learning also provides the tools for testing cognitive theories. In order to test a theory, we need to realise the proposed information-processing system at scale, so as to be able to assess its feasibility and emergent behaviours. Deep learning allows us to scale up from principles and circuit models to end-to-end trainable models capable of performing complex tasks. There are many levels at which cognitive neuroscientists can use deep learning in their work, from inspiring theories to serving as full computational models. Ongoing advances in deep learning bring us closer to understanding how cognition and perception may be implemented in the brainthe grand challenge at the core of cognitive neuroscience. Introduction: neuro-inspired AI and AI-inspired neuroscienceTo explain how brains reason, remember, perceive, and act, theories must bridge from biology to behaviour. Rigorous explanation requires models that use neurobiologically plausible components to im-

show abstract

“…Equilibrium propagation [36] is a new framework for machine learning, where the prediction and the objective function are defined implicitly through an energy function of the data and the parameters of the model, rather than explicitly as in a feedforward net. The energy function F (θ, β, s, v) is defined to model all interactions within the system and the actions with the external world of the system, where θ is the parameter to be learned, β is a parameter to control the level of the influence of the external world, v is the state of the external world(input and expected output) and s is the state of the system.…”

Section: F Geometry Of Equilibrium Propagationmentioning

confidence: 99%

“…Another observation is that the equilibrium propagation is not only a framework for training the so called implicitly defined deep learning systems as described in [36], in fact it's also potentially a framework to optimize the structure of the explicitly defined deep learning systems as well.…”

Section: • Gradientmentioning

confidence: 99%

How deep learning works --The geometry of deep learning

Dong,

Wu,

Zhou

2017

Preprint

View full text Add to dashboard Cite

Why and how that deep learning works well on different tasks remains a mystery from a theoretical perspective. In this paper we draw a geometric picture of the deep learning system by finding its analogies with two existing geometric structures, the geometry of quantum computations and the geometry of the diffeomorphic template matching. In this framework, we give the geometric structures of different deep learning systems including convolutional neural networks, residual networks, recursive neural networks, recurrent neural networks and the equilibrium prapagation framework. We can also analysis the relationship between the geometrical structures and their performance of different networks in an algorithmic level so that the geometric framework may guide the design of the structures and algorithms of deep learning systems.

show abstract

Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation

Cited by 5 publications

References 0 publications

Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation

Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation

Deep Learning for Cognitive Neuroscience

How deep learning works --The geometry of deep learning

Contact Info

Product

Resources

About