Technical report: supervised training of convolutional spiking neural networks with PyTorch

Zimmer, Romain; Pellegrini, Thomas; Singh, Srisht Fateh; Masquelier, Timothée

doi:10.48550/arxiv.1911.10124

Cited by 7 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the non-differentiability of the thresholding activation function in spiking neurons, it is not convenient to apply backpropagation and gradient descents to SNNs. Various solutions are proposed to tackle this problem including computing gradients with respect to the spike rates instead of single spikes [52,53,54,55], using differentiable smoothed spike functions [56], surrogate gradients for the threshold function in the backward pass [57,58,59,60,61,62], and transfer learning by sharing weights between the SNN and an ANN [49,63]. In another approach, known as latency learning, the neuron's activity is defined based on the firing time of its first spike, therefore, they do not need to compute the gradient of the thresholding function.…”

Section: Discussionmentioning

confidence: 99%

BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning

Kheradpisheh,

Mirsadeghi,

Masquelier

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We recently proposed the S4NN algorithm, essentially an adaptation of backpropagation to multilayer spiking neural networks that use simple nonleaky integrate-and-fire neurons and a form of temporal coding known as time-to-first-spike coding. With this coding scheme, neurons fire at most once per stimulus, but the firing order carries information. Here, we introduce BS4NN, a modification of S4NN in which the synaptic weights are constrained to be binary (+1 or -1), in order to decrease memory and computation footprints. This was done using two sets of weights: firstly, real-valued weights, updated by gradient descent, and used in the backward pass of backpropagation, and secondly, their signs, used in the forward pass. Similar strategies have been used to train (non-spiking) binarized neural networks. The main difference is that BS4NN operates in the time domain: spikes are propagated sequentially, and different neurons may reach their threshold at different times, which increases computational power. We validated BS4NN on two popular benchmarks, MNIST and Fashion MNIST, and obtained state-of-the-art accuracies for this sort of networks (97.0% and 87.3% respectively) with a negligible accuracy drop with re- *

show abstract

Section: Discussionmentioning

confidence: 99%

BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning

Kheradpisheh,

Mirsadeghi,

Masquelier

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Before the audio signals move into the model, they should be preprocessed. For preprocessing, we use a similar approach as Zimmer et al ( 2019 ). We remove extra information from the audio signal, the model becomes simpler and also it is more robust to noise.…”

Section: Methodsmentioning

confidence: 99%

E-prop on SpiNNaker 2: Exploring online learning in spiking RNNs on neuromorphic hardware

et al. 2022

View full text Add to dashboard Cite

IntroductionIn recent years, the application of deep learning models at the edge has gained attention. Typically, artificial neural networks (ANNs) are trained on graphics processing units (GPUs) and optimized for efficient execution on edge devices. Training ANNs directly at the edge is the next step with many applications such as the adaptation of models to specific situations like changes in environmental settings or optimization for individuals, e.g., optimization for speakers for speech processing. Also, local training can preserve privacy. Over the last few years, many algorithms have been developed to reduce memory footprint and computation.MethodsA specific challenge to train recurrent neural networks (RNNs) for processing sequential data is the need for the Back Propagation Through Time (BPTT) algorithm to store the network state of all time steps. This limitation is resolved by the biologically-inspired E-prop approach for training Spiking Recurrent Neural Networks (SRNNs). We implement the E-prop algorithm on a prototype of the SpiNNaker 2 neuromorphic system. A parallelization strategy is developed to split and train networks on the ARM cores of SpiNNaker 2 to make efficient use of both memory and compute resources. We trained an SRNN from scratch on SpiNNaker 2 in real-time on the Google Speech Command dataset for keyword spotting.ResultWe achieved an accuracy of 91.12% while requiring only 680 KB of memory for training the network with 25 K weights. Compared to other spiking neural networks with equal or better accuracy, our work is significantly more memory-efficient.DiscussionIn addition, we performed a memory and time profiling of the E-prop algorithm. This is used on the one hand to discuss whether E-prop or BPTT is better suited for training a model at the edge and on the other hand to explore architecture modifications to SpiNNaker 2 to speed up online learning. Finally, energy estimations predict that the SRNN can be trained on SpiNNaker2 with 12 times less energy than using a NVIDIA V100 GPU.

show abstract

“…The differential equations of LIF models can be approximated by linear recurrent equations in discrete time. Introducing a reset term U R i [n] for the potential, the neuron dynamics can now be fully described by the following equations [13].…”

Section: Overview Of the Leaky Integrate-and-fire Model (Lif)mentioning

confidence: 99%

Low-Activity Supervised Convolutional Spiking Neural Networks Applied to Speech Commands Recognition

Pellegrini

Zimmer

Masquelier

2021

2021 IEEE Spoken Language Technology Workshop (SLT)

Self Cite

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are the current stateof-the-art models in many speech related tasks. There is a growing interest, though, for more biologically realistic, hardware friendly and energy efficient models, named Spiking Neural Networks (SNNs). Recently, it has been shown that SNNs can be trained efficiently, in a supervised manner, using backpropagation with a surrogate gradient trick. In this work, we report speech command (SC) recognition experiments using supervised SNNs. We explored the Leaky-Integrate-Fire (LIF) neuron model for this task, and show that a model comprised of stacked dilated convolution spiking layers can reach an error rate very close to standard DNNs on the Google SC v1 dataset: 5.5%, while keeping a very sparse spiking activity, below 5%, thank to a new regularization term. We also show that modeling the leakage of the neuron membrane potential is useful, since the LIF model outperformed its non-leaky model counterpart significantly.

show abstract

Technical report: supervised training of convolutional spiking neural networks with PyTorch

Cited by 7 publications

References 19 publications

BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning

BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning

E-prop on SpiNNaker 2: Exploring online learning in spiking RNNs on neuromorphic hardware

Low-Activity Supervised Convolutional Spiking Neural Networks Applied to Speech Commands Recognition

Contact Info

Product

Resources

About