Equivalence of Equilibrium Propagation and Recurrent Backpropagation

Scellier, Benjamin; Bengio, Yoshua

doi:10.48550/arxiv.1711.08416

Cited by 6 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this appendix we study the dynamics itself in the second phase when the state of the network moves from the free fixed point s 0 θ to the weakly clamped fixed point s β θ . The result established in this section is a straightforward generalization of the result proved in Scellier and Bengio [2017b].…”

Section: B Link To Recurrent Backpropagationsupporting

confidence: 60%

“…and by Theorems 3 and 4 we get S t = S t and Θ t = Θ t . This result was stated and proved in Scellier and Bengio [2017b].…”

Section: Appendixmentioning

confidence: 63%

“…A more detailed analysis of the second phase is carried out in Appendix B and a connection to the recurrent backpropagation algorithm [Almeida, 1987, Pineda, 1987 is established, following the ideas of Scellier and Bengio [2017b].…”

Section: Backpropagation Of Error Signalsmentioning

confidence: 99%

See 2 more Smart Citations

Generalization of Equilibrium Propagation to Vector Field Dynamics

Scellier¹,

Goyal²,

Binas³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

The biological plausibility of the backpropagation algorithm has long been doubted by neuroscientists. Two major reasons are that neurons would need to send two different types of signal in the forward and backward phases, and that pairs of neurons would need to communicate through symmetric bidirectional connections. We present a simple two-phase learning procedure for fixed point recurrent networks that addresses both these issues. In our model, neurons perform leaky integration and synaptic weights are updated through a local mechanism. Our learning method generalizes Equilibrium Propagation to vector field dynamics, relaxing the requirement of an energy function. As a consequence of this generalization, the algorithm does not compute the true gradient of the objective function, but rather approximates it at a precision which is proven to be directly related to the degree of symmetry of the feedforward and feedback weights. We show experimentally that our algorithm optimizes the objective function.

show abstract

Section: B Link To Recurrent Backpropagationsupporting

confidence: 60%

“…and by Theorems 3 and 4 we get S t = S t and Θ t = Θ t . This result was stated and proved in Scellier and Bengio [2017b].…”

Section: Appendixmentioning

confidence: 63%

See 1 more Smart Citation

Generalization of Equilibrium Propagation to Vector Field Dynamics

Scellier¹,

Goyal²,

Binas³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…RBP is naturally applicable here and demonstrated to save both computation time and memory. A recent investigation (Scellier & Bengio, 2017a) shows that RBP is related to equilibrium propagation (Scellier & Bengio, 2017b) which is motivated from the perspective of biological plausibility. Another recent related work in deep learning is OptNet (Amos & Kolter, 2017) where the gradient of the optimized solution of a quadratic programming problem w.r.t.…”

Section: Related Workmentioning

confidence: 99%

Reviving and Improving Recurrent Back-Propagation

Liao,

Xiong,

Fetaya

et al. 2018

Preprint

View full text Add to dashboard Cite

In this paper, we revisit the recurrent backpropagation (RBP) algorithm (Almeida, 1987;Pineda, 1987), discuss the conditions under which it applies as well as how to satisfy them in deep neural networks. We show that RBP can be unstable and propose two variants based on conjugate gradient on the normal equations (CG-RBP) and Neumann series (Neumann-RBP). We further investigate the relationship between Neumann-RBP and back propagation through time (BPTT) and its truncated version (TBPTT). Our Neumann-RBP has the same time complexity as TBPTT but only requires constant memory, whereas TBPTT's memory cost scales linearly with the number of truncation steps. We examine all RBP variants along with BPTT and TBPTT in three different application domains: associative memory with continuous Hopfield networks, document classification in citation networks using graph neural networks and hyperparameter optimization for fully connected networks. All experiments demonstrate that RBPs, especially the Neumann-RBP variant, are efficient and effective for optimizing convergent recurrent neural networks. Code is released at: https://github.com/ lrjconan/RBP.

show abstract

“…Only recently have biological plausible proposals been made to adapt backpropagation to spiking and continuous time neural networks with reasonable success. Local contrastive Hebbian learning in energy based models are one class of proposals that have shown a level of equivalence with recurrent backpropagation [6,7,8]. In these learning algorithms, the networks minimizes the difference between the fixed points when running freely and when clamped to an external input for error correction.…”

Section: Introductionmentioning

confidence: 99%

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Kohan,

Rietman,

Siegelmann

2018

Preprint

View full text Add to dashboard Cite

We introduce Error Forward-Propagation, a biologically plausible mechanism to propagate error feedback forward through the network. Architectural constraints on connectivity are virtually eliminated for error feedback in the brain; systematic backward connectivity is not used or needed to deliver error feedback. Feedback as a means of assigning credit to neurons earlier in the forward pathway for their contribution to the final output is thought to be used in learning in the brain. How the brain solves the credit assignment problem is unclear. In machine learning, error backpropagation is a highly successful mechanism for credit assignment in deep multilayered networks. Backpropagation requires symmetric reciprocal connectivity for every neuron. From a biological perspective, there is no evidence of such an architectural constraint, which makes backpropagation implausible for learning in the brain. This architectural constraint is reduced with the use of random feedback weights. Models using random feedback weights require backward connectivity patterns for every neuron, but avoid symmetric weights and reciprocal connections. In this paper, we practically remove this architectural constraint, requiring only a backward loop connection for effective error feedback. We propose reusing the forward connections to deliver the error feedback by feeding the outputs into the input receiving layer. This mechanism, Error Forward-Propagation, is a plausible basis for how error feedback occurs deep in the brain independent of and yet in support of the functionality underlying intricate network architectures. We show experimentally that recurrent neural networks with two and three hidden layers can be trained using Error Forward-Propagation on the MNIST and Fashion MNIST datasets, achieving 1.90% and 11% generalization errors respectively.

show abstract

Equivalence of Equilibrium Propagation and Recurrent Backpropagation

Cited by 6 publications

References 0 publications

Generalization of Equilibrium Propagation to Vector Field Dynamics

Generalization of Equilibrium Propagation to Vector Field Dynamics

Reviving and Improving Recurrent Back-Propagation

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Contact Info

Product

Resources

About