Supervised Learning Based on Temporal Coding in Spiking Neural Networks

Abstract

Gradient descent training techniques are remarkably successful in training analogvalued artificial neural networks (ANNs). Such training techniques, however, do not transfer easily to spiking networks due to the spike generation hard nonlinearity and the discrete nature of spike communication. We show that in a feedforward spiking network that uses a temporal coding scheme where information is encoded in spike times instead of spike rates, the network input-output relation is differentiable almost everywhere. Moreover, this relation is piece-wise linear after a transformation of variables. Methods for training ANNs thus carry directly to the training of such spiking networks as we show when training on the permutation invariant MNIST task. In contrast to rate-based spiking networks that are often used to approximate the behavior of ANNs, the networks we present spike much more sparsely and their behavior can not be directly approximated by conventional ANNs. Our results highlight a new approach for controlling the behavior of spiking networks with realistic temporal dynamics, opening up the potential for using these networks to process spike patterns with complex temporal information.

Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines

“…More recently, (Mostafa, 2016) used a temporal coding scheme where information is encoded in spike times instead of spike rates and the dynamics are cast in a differentiable form. As a result, the network can be trained using standard gradient descent to achieve very accurate, sparse and power-efficient classification.…”

Event-driven random backpropagation: Enabling neuromorphic deep learning machines

“…More recently, (Mostafa, 2016) used a temporal coding scheme where information is encoded in spike times instead of spike rates and the dynamics are cast in a differentiable form. As a result, the network can be trained using standard gradient descent to achieve very accurate, sparse and power-efficient classification.…”

Spiking Neural Networks Trained With Backpropagation for Low Power Neuromorphic Implementation of Voice Activity Detection

“…We addressed the connectivity limitations of neuromorphic chips by testing the lottery ticket hypothesis [19] with positive results achieving even less energy consumption. Previous studies [16,23,32] applied similar models to simple image datasets, where the features precise values are not necessary to solve the task, namely the task can be solved by using only black and white [32]. The novelty of the proposed work lies in showing how these models are able to process more complex continuous features such as Mel filterbank coefficients in a real-world application scenario, highlighting the power of the method and the high level of sparsity these features can be represented with, in contrast with ANN to SNN conversion methods that rely on spike rates to convey information [18] and to the spiking VAD solution proposed in [17] at 26.1 mW.…”