Training End-to-End Analog Neural Networks with Equilibrium Propagation

Abstract

We introduce a principled method to train end-to-end analog neural networks by stochastic gradient descent. In these analog neural networks, the weights to be adjusted are implemented by the conductances of programmable resistive devices such as memristors [Chua, 1971], and the nonlinear transfer functions (or 'activation functions') are implemented by nonlinear components such as diodes. We show mathematically that a class of analog neural networks (called nonlinear resistive networks) are energy-based models: they possess an energy function as a consequence of Kirchhoff's laws governing electrical circuits. This property enables us to train them using the Equilibrium Propagation framework [Scellier and Bengio, 2017]. Our update rule for each conductance, which is local and relies solely on the voltage drop across the corresponding resistor, is shown to compute the gradient of the loss function. Our numerical simulations, which use the SPICEbased Spectre simulation framework to simulate the dynamics of electrical circuits, demonstrate training on the MNIST classification task, performing comparably or better than equivalent-size software-based neural networks. Our work can guide the development of a new generation of ultra-fast, compact and low-power neural networks supporting on-chip learning. * Currently at Google. Preprint. Under review.

A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation

“…We will now separate the Lagrangian into sets of terms which will aid in our analysis. We assume the network takes the form of a deep neural network, similar to our previous work [37], with layers of nonlinear, dynamic subcircuits (neurons) separated by fully connected layers of linear resistors (synapses).…”

Monotone RLC Circuits

“…The fundamental status of the memristor, however, has remained contentious [45]. Since the 2008 discovery, interest in the memristor has surged, primarily due to its potential in neuromorphic computers and analog neural networks [27], [29]- [32]. We note that the invention of the memristor by Chua [44] follows the first paper of monotone circuits [16] by a decade, but that, to the best of the authors' knowledge, all the system-theoretic investigations of the memristor have focused on passivity rather than monotonicity, although Theorem 1 of Chua [44] can be interpreted as showing that monotonicity is equivalent to passivity for a memristor.…”

“…This line of research exploits the algorithmic significance of maximal monotonicity, but has progressively become detached from its physical significance. On the other hand, recent years have witnessed a surge of interest in the physical significance of memristive nonlinear circuits [26]- [32], but with little emphasis on algorithms and system analysis.…”

Physical learning beyond the quasistatic limit

“…In order for such physical networks to learn on their own, they cannot minimize an arbitrary cost function by gradient descent since that is a global process that requires knowing all the microscopic details at once, carrying out the global computation of gradient descent, and then manipulating networks at the microscopic (node or edge) level. Rather, approaches such as contrastive learning (1)(2)(3)(4), equilibrium propagation (5,6), directed aging (7)(8)(9)(10)(11)(12) and coupled learning (13,14) use local rules, in which learning degrees of freedom (e.g. the conductances of edges in electrical networks of variable resistors) respond to physical degrees of freedom (e.g.…”