Temporal Backpropagation for Spiking Neural Networks with One Spike per Neuron

Abstract

We propose a new supervised learning rule for multilayer spiking neural networks (SNN) that use a form of temporal coding known as rank-ordercoding. With this coding scheme, all neurons fire exactly one spike per stimulus, but the firing order carries information. In particular, in the readout layer the first neuron to fire determines the class of the stimulus. We derive a new learning rule for this sort of network, termed S4NN, akin to traditional error backpropagation, yet based on latencies. We show how approximate error gradients can be computed backward in a feedforward network with an arbitrary number of layers. This approach reaches state-of-the-art performance with SNNs: test accuracy of 97.4% for the MNIST dataset, and of 99.2% for the Caltech Face/Motorbike dataset. Yet the neuron model we use, non-leaky integrate-and-fire, are simpler and more hardware friendly than the one used in all previous similar proposals.

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

“…However, it is known that rate coding does not allow the network to use spike-times precisely which can, in turn, enable an SNN to encode more information or process information rapidly. Supervised learning based SNNs using latency-based coding scheme is a good way to decrease the energy consumption, compared to the rate-coding method (Mostafa, 2017 ; Comsa et al, 2019 ; Kheradpisheh and Masquelier, 2019 ; Zhou et al, 2019 ). In latency-based coding, pixel intensity is represented by the ascending order of incoming spikes, wherein higher intensity fires an earlier spike and vice-versa.…”

“…As a result, more salient information about a feature is encoded as an earlier spike in the corresponding neuron leading to overall sparser activity in an SNN. Furthermore, the inference latency (or overall time steps required to process an input) can drastically decrease to few 10 time steps with appropriate learning methods instead of the usual 50–100 time steps incurred in rate coding schemes for AGD training (Mostafa, 2017 ; Comsa et al, 2019 ; Kheradpisheh and Masquelier, 2019 ; Roy et al, 2019 ; Zhou et al, 2019 ).…”

“…With these new additions, Zhou et al present the best results obtained so far on a purely temporal based backpropagation learning scheme for CIFAR10 (80.5%). In addition, other recent works (Comsa et al, 2019;Kheradpisheh and Masquelier, 2019) have also shown impressive and promising results on the use of first-to-spike latency coding in realizing the energy-efficiency and inference latency benefits with SNN implementations. However, the framework (including the network architecture and learning rule used to perform spike-based backpropagation) is very different in each work.…”

Visualizing a joint future of neuroscience and neuromorphic engineering

“…Another approach assumes that each neuron spikes precisely once in a given time period and computes gradients with respect to these spike times ( Figure 1B). Kheradpisheh and Masquelier (2020) showed that it yields state-of-the-art accuracy for spike latency-encoded versions of MNIST and fashion MNIST. In similar work, Comsa et al, 2019 not only achieved competitive (A) Instilling functions at the network level requires hidden neurons, which are neither connected to the input nor the network's output, to reduce their contribution to errors at the output level.…”

“…The method assumes extreme sparseness of spiking because every neuron emits, at most, one spike. This representation allows efficient eventdriven algorithms in which time represents itself, which translates algorithmically into a small memory footprint and lowpower computation at the network level (Kheradpisheh and Masquelier 2020;Gö ltz et al, 2019). Similar to a binary neural network, all processing occurs as a single volley of spikes propagates through the network.…”