Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Yin, Shihui; Venkataramanaiah, Shreyas Kolala; Chen, Gregory K.; Krishnamurthy, Ram; Cao, Yu; Chakrabarti, Chaitali; Seo, Jae-sun

doi:10.1109/biocas.2017.8325230

Cited by 51 publications

(57 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, to compare with existing onchip works, Yin et al (2017) presented a new BP based training algorithm for discrete-time SNNs by using a LIF neuron model with a gradient estimator. This paper introduced a ReLU-like gradient estimation method to avoid the zero-gradient issue in conventional SNNs using LIF neurons.…”

Section: Discussionmentioning

confidence: 99%

“…However, as the experiment results in Yin et al (2017) are based on off-chip training, we guess that this new BP algorithm still suffers from an efficient on-chip training method. We think that the main difference of Yin et al (2017) and our work is that Yin et al (2017) proposed a new BPbased learning algorithm while our work proposes a new BPlike learning algorithm (i.e., DFA) based on a state-of-the-art BP algorithm and efficiently implement it in hardware while delivering competitive accuracy. As a small scale low-power accelerator, Zheng and Pinaki (2018) proposed a hardwarefriendly STDP on-chip training algorithm.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Spike-Train Level Direct Feedback Alignment: Sidestepping Backpropagation for On-Chip Training of Spiking Neural Nets

Lee

Zhang

et al. 2020

Front. Neurosci.

View full text Add to dashboard Cite

Spiking neural networks (SNNs) present a promising computing model and enable bio-plausible information processing and event-driven based ultra-low power neuromorphic hardware. However, training SNNs to reach the same performances of conventional deep artificial neural networks (ANNs), particularly with error backpropagation (BP) algorithms, poses a significant challenge due to inherent complex dynamics and non-differentiable spike activities of spiking neurons. In this paper, we present the first study on realizing competitive spike-train level backpropagation (BP) like algorithms to enable on-chip training of SNNs. We propose a novel spike-train level direct feedback alignment (ST-DFA) algorithm, which is much more bio-plausible and hardware friendly than BP. Algorithm and hardware co-optimization and efficient online neural signal computation are explored for on-chip implementation of ST-DFA. On the Xilinx ZC706 FPGA board, the proposed hardware-efficient ST-DFA shows excellent performance vs. overhead tradeoffs for real-world speech and image classification applications. SNN neural processors with on-chip ST-DFA training show competitive classification accuracy of 96.27% for the MNIST dataset with 4× input resolution reduction and 84.88% for the challenging 16-speaker TI46 speech corpus, respectively. Compared to the hardware implementation of the state-of-the-art BP algorithm HM2-BP, the design of the proposed ST-DFA reduces functional resources by 76.7% and backward training latency by 31.6% while gracefully trading off classification performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Spike-Train Level Direct Feedback Alignment: Sidestepping Backpropagation for On-Chip Training of Spiking Neural Nets

Lee

Zhang

et al. 2020

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Designs targeting very large scale applications usually implement a large number of neurocores to maximize parallelization at chip level [15,38,49,122,148,160]. A large amount of publications report various neurocore organizations with designs optimized in accordance with the characteristics of the network topology to be mapped to the hardware [28,91,130,164,193], or with a targeted application [34,38,103,148,153,192,194,201].…”

Section: Core Organization For Low Power Spiking Neural Network Evalumentioning

confidence: 99%

Spiking Neural Networks Hardware Implementations and Challenges

Bouvier

Valentian

Mesquida

et al. 2019

J. Emerg. Technol. Comput. Syst.

181

View full text Add to dashboard Cite

Neuromorphic computing is henceforth a major research field for both academic and industrial actors. As opposed to Von Neumann machines, brain-inspired processors aim at bringing closer the memory and the computational elements to efficiently evaluate machine-learning algorithms. Recently, Spiking Neural Networks, a generation of cognitive algorithms employing computational primitives mimicking neuron and synapse operational principles, have become an important part of deep learning. They are expected to improve the computational performance and efficiency of neural networks, but are best suited for hardware able to support their temporal dynamics. In this survey, we present the state of the art of hardware implementations of spiking neural networks and the current trends in algorithm elaboration from model selection to training mechanisms. The scope of existing solutions is extensive; we thus present the general framework and study on a case-by-case basis the relevant particularities. We describe the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level and discuss their related advantages and challenges.

show abstract

“…This fact is directly explored in some digital spiking neural network implementations (Yin et al., 2017). Furthermore, the statefulness of the neurons and the filtering of their inputs is consistent with recurrent neural networks, even when the network is of the feedforward type.…”

Section: Distilling Machine Learning and Neuroscience For Neuromorphimentioning

confidence: 99%

Data and Power Efficient Intelligence with Neuromorphic Learning Machines

Neftci

2018

iScience

View full text Add to dashboard Cite

The success of deep networks and recent industry involvement in brain-inspired computing is igniting a widespread interest in neuromorphic hardware that emulates the biological processes of the brain on an electronic substrate. This review explores interdisciplinary approaches anchored in machine learning theory that enable the applicability of neuromorphic technologies to real-world, human-centric tasks. We find that (1) recent work in binary deep networks and approximate gradient descent learning are strikingly compatible with a neuromorphic substrate; (2) where real-time adaptability and autonomy are necessary, neuromorphic technologies can achieve significant advantages over main-stream ones; and (3) challenges in memory technologies, compounded by a tradition of bottom-up approaches in the field, block the road to major breakthroughs. We suggest that a neuromorphic learning framework, tuned specifically for the spatial and temporal constraints of the neuromorphic substrate, will help guiding hardware algorithm co-design and deploying neuromorphic hardware for proactive learning of real-world data.

show abstract

Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Cited by 51 publications

References 20 publications

Spike-Train Level Direct Feedback Alignment: Sidestepping Backpropagation for On-Chip Training of Spiking Neural Nets

Spike-Train Level Direct Feedback Alignment: Sidestepping Backpropagation for On-Chip Training of Spiking Neural Nets

Spiking Neural Networks Hardware Implementations and Challenges

Data and Power Efficient Intelligence with Neuromorphic Learning Machines

Contact Info

Product

Resources

About