A Soft-Pruning Method Applied During Training of Spiking Neural Networks for In-memory Computing Applications

Shi, Yuhan; Nguyen, Leon; Oh, Sangheon; Liu, Xin; Kuzum, Duygu

doi:10.3389/fnins.2019.00405

Cited by 38 publications

(43 citation statements)

References 52 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The software simulation is programmed in Python language and runs sequentially in a single process. The runtime decreases with the increasing pruning percentage (decreasing connectivity), which proves that the proposed online pruning method is able to shorten the network Online soft weight pruning Shi et al, 2019 12.23%/9.63% 27%/29% 0%/0%…”

Section: Implementation Overheadmentioning

confidence: 66%

“…An online soft weight pruning method for unsupervised SNNs was reported in Shi et al (2019) . Unlike conventional pruning methods, instead of removing the pruned weights, this method sets the pruned weights constant at the lowest possible weight value or the current value and stops updating them for the rest of the training process.…”

Section: Comparisons and Discussionmentioning

confidence: 99%

“…Pruning is commonly applied after training, which is suitable for improving the energy efficiency of inference systems with offline training. Pruning while training technique has been proved to be very useful in SNNs to improve online learning systems that can learn and infer the real-world information (Rathi et al, 2019;Shi et al, 2019). However, in the previously reported online pruning methods, pruning was conducted with a constant threshold for all the synaptic weights during training.…”

Section: Overview Of the Proposed Pruning Methodsmentioning

confidence: 99%

“…However, the soft pruning method brings less benefit to the computational cost compared with the adaptive pruning method. Figure 14B shows that it contributes to less reduction in training SOPs/image and no reduction in inference FIGURE 14 | Comparison with the online soft weight pruning method adopted from Shi et al (2019) in the SNN trained on MNIST dataset with 100 excitatory neurons. (A) Accuracy and (B) normalized SOPs/image change with unpruned weights percentage.…”

Section: Comparison With Prior Workmentioning

confidence: 99%

“…While weight pruning has been widely applied in different ANNs, the benefits that weight pruning could provide for SNNs have yet to be explored. Limited works have reported applying weight pruning in SNNs so far ( Iglesias et al, 2005 ; Rathi et al, 2019 ; Shi et al, 2019 ). Rathi et al proposed a spike-timing-dependent plasticity (STDP) based online synaptic pruning method, which sets non-critical weights to zero during the training phase and removes the weights below a certain threshold at the end of training ( Rathi et al, 2019 ).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems

et al. 2020

View full text Add to dashboard Cite

To tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications.

show abstract

Section: Implementation Overheadmentioning

confidence: 66%

Section: Comparisons and Discussionmentioning

confidence: 99%

Section: Overview Of the Proposed Pruning Methodsmentioning

confidence: 99%

Section: Comparison With Prior Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems

et al. 2020

View full text Add to dashboard Cite

show abstract

Progressive compression and weight reinforcement for spiking neural networks

Elbez

Benhaoua

Devienne

et al. 2022

Concurrency and Computation

View full text Add to dashboard Cite

Neuromorphic architectures are one of the most promising architectures to significantly reduce the energy consumption of tomorrow's computers. These architectures are inspired by the behaviour of the brain at a fairly precise level and consist of artificial Spiking Neural Networks (SNNs). To optimise the implementation of these architectures, we propose in this paper a novel progressive network compression and reinforcement technique based on two functions, progressive pruning and dynamic synaptic weight reinforcement used after each training batch. The proposed approach delivers a highly compressed network (up to 75 % of compression rate) while preserving the network performance when tested with MNIST.

show abstract

An overview memristor based hardware accelerators for deep neural network

Gökgöz,

Gül,

Aydın

2024

Concurrency and Computation

View full text Add to dashboard Cite

The prevalence of artificial intelligence applications using artificial neural network architectures for functions such as natural language processing, text prediction, object detection, speech, and image recognition has significantly increased in today's world. The computational functions performed by artificial neural networks in classical applications require intensive and large‐scale data movement between memory and processing units. Various software and hardware efforts are being made to perform these operations more efficiently. Despite these efforts, latency in data traffic and the substantial amount of energy consumed in data processing emerge as bottleneck disadvantages of the Von Neumann architecture. To overcome this bottleneck problem, it is necessary to develop hardware units specific to artificial intelligence applications. For this purpose, neuro‐inspired computing chips are believed to provide an effective approach by designing and integrating a set of features inspired by neurobiological systems at the hardware level to address the problems arising in artificial intelligence applications. The most notable among these approaches is memristor‐based neuromorphic computing systems. Memristors are seen as promising devices for hardware‐level improvement in terms of speed and energy because they possess non‐volatile memory and exhibit analog behavior. They enable effective storage and processing of synaptic weights, offering solutions for hardware‐level development. Taking into account these advantages of memristors, this study examines the research conducted on artificial neural networks and hardware that can directly perform deep learning functions and mimic the biological brain, which is different from classical systems in today's context.

show abstract

A Soft-Pruning Method Applied During Training of Spiking Neural Networks for In-memory Computing Applications

Cited by 38 publications

References 52 publications

Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems

Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems

Progressive compression and weight reinforcement for spiking neural networks

An overview memristor based hardware accelerators for deep neural network

Contact Info

Product

Resources

About