Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

Sengupta, Abhronil; Ye, Yuting; Wang, Robert; Liu, Chiao; Roy, Kaushik

doi:10.48550/arxiv.1802.02627

Cited by 6 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is important to set the neuronal firing thresholds sufficiently high so that each spiking neuron can closely resemble ANN activation without loss of information. In the literature, several methods have been proposed [4,8,14,38,35] for balancing appropriate ratios between neuronal thresholds and synaptic weights of spiking neuron in the case of ANN-SNN conversion. In this paper, we compare various aspects of our direct-spike trained models with one recent work [38], which proposed a near-lossless ANN-SNN conversion scheme for deep network architectures.…”

Section: Ann-snn Conversion Schemementioning

confidence: 99%

“…This mechanism enables event-based and asynchronous computations across the layers on spiking systems, which makes it naturally suitable for ultra-low power computing. Furthermore, recent works [38,35] have shown that these properties make SNNs significantly more attractive for deeper networks in the case of hardware implementation. This is because the spike signals become significantly sparser as the layer goes deeper, such that the number of required computations significantly reduces.…”

Section: Introductionmentioning

confidence: 99%

“…The general training strategy of SNNs can be categorized in two ways -ANN-SNN conversion and direct spike-based training. First, there are studies which have successfully deployed the ANN-SNN conversion technique that transforms off-line trained ANN to SNN for efficient event-based inference [4,8,14,38,35]. The main objective of ANN-SNN conversion scheme is to leverage the state-of-the-art ANN training techniques, so that the transformed networks can mimic the competitive classification performances of the ANNs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

et al. 2020

Self Cite

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow spiking network architectures have limited capacity for expressing complex representations, while training a very deep spiking network has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-line trained deep Artificial Neural Networks (ANNs) to SNNs. However, ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous and non-differentiable nature of spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for leaky behavior of LIF neuron. This method enables training of deep convolutional SNNs with input spike events using spike-based backpropagation algorithm. Our experiments show the effectiveness of the proposed spike-based learning strategy on state-of-the-art deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain. and show remarkable results, which occasionally outperform human-level performance [20,13,40]. To that effect, deploying deep learning is becoming necessary not only on large-scale computers, but also on edge devices (e.g. phone, tablet, smart watch, robot etc.). However, the ever-growing complexity of the state-of-the-art deep neural networks together with the explosion in the amount of data to be processed, place significant energy demands on current computing platforms. For example, a deep ANN model requires unprecedented amount of computing hardware resources that often requires huge computing power of cloud servers and significant amount of time to train.Spiking Neural Network (SNN) is one of the leading candidates for overcoming the constraints of neural computing and to efficiently harness the machine learning algorithm in real-life (or mobile) applications [28,5]. The concepts of SNN, which is often regarded as the 3 rd generation neural network [27], are inspired by biologically plausible Leaky Integrate and Fire (LIF) spiking neuron models [6] that can efficiently process spatio-temporal information. The LIF neuron model is characterized by the internal state, called membrane potential, that integrates the inputs over time and generates an output spike whenever it overcomes the neuronal firing threshold. This mechanism enables event-based and asynchronous computations across the layers on spiking systems, which makes it naturally suitable for ultra-low power computing. Furthermore, recent works [38,35] have shown that these properties make SNNs significantly more attractive for deeper networks in the case of h...

show abstract

Section: Ann-snn Conversion Schemementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…SNNs were proven to be effective in several problems but it remained less competitive compared to CNNs. Recently (Sengupta et al, 2018) demonstrated that a deep SNN can achieve better accuracy than CNN on a challenging dataset ImageNet. A detailed overview of deep learning in SNN is discussed in (Tavanaei et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

Exploring Deep Spiking Neural Networks for Automated Driving Applications

Mohapatra

Gotzig

Yogamani

et al. 2019

Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application

View full text Add to dashboard Cite

Neural networks have become the standard model for various computer vision tasks in automated driving including semantic segmentation, moving object detection, depth estimation, visual odometry, etc. The main flavors of neural networks which are used commonly are convolutional (CNN) and recurrent (RNN). In spite of rapid progress in embedded processors, power consumption and cost is still a bottleneck. Spiking Neural Networks (SNNs) are gradually progressing to achieve low-power event-driven hardware architecture which has a potential for high efficiency. In this paper, we explore the role of deep spiking neural networks (SNN) for automated driving applications. We provide an overview of progress on SNN and argue how it can be a good fit for automated driving applications.

show abstract

“…Unlike ANNs, SNNs are hybrid digital-analog machines that make use of the temporal dimension, not just as a neutral substrate for computing, but as a means to encode and process information [7]. Training methods for SNNs typically assume deterministic non-linear dynamic models for the spiking neurons, and are either motivated by biological plausibility, such as the spiketiming-dependent plasticity (STDP) rule [5], [8], or by an attempt to mimic the operation of ANNs and associated learning rules (see, e.g., [9] and references therein). Deterministic models are known to be limited in their expressive power, especially as it pertains prior domain knowledge, uncertainty, and definition of generic queries and tasks.…”

Section: Introductionmentioning

confidence: 99%

Adversarial Training for Probabilistic Spiking Neural Networks

Bagheri

Simeone

Rajendran

2018

2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)

View full text Add to dashboard Cite

Classifiers trained using conventional empirical risk minimization or maximum likelihood methods are known to suffer dramatic performance degradations when tested over examples adversarially selected based on knowledge of the classifier's decision rule. Due to the prominence of Artificial Neural Networks (ANNs) as classifiers, their sensitivity to adversarial examples, as well as robust training schemes, have been recently the subject of intense investigation. In this paper, for the first time, the sensitivity of spiking neural networks (SNNs), or third-generation neural networks, to adversarial examples is studied. The study considers rate and time encoding, as well as rate and first-to-spike decoding. Furthermore, a robust training mechanism is proposed that is demonstrated to enhance the performance of SNNs under white-box attacks.

show abstract

Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

Cited by 6 publications

References 0 publications

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

Exploring Deep Spiking Neural Networks for Automated Driving Applications

Adversarial Training for Probabilistic Spiking Neural Networks

Contact Info

Product

Resources

About