Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Kim, Youngeun; Li, Yuhang; Park, Hyoungseob; Venkatesha, Yeshwanth; Yin, Ruokai; Panda, Priyadarshini

doi:10.1007/978-3-031-19775-8_7

Cited by 18 publications

(9 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Shi et al (2019) prune weight connections during training with a soft mask. Recently, deeper SNNs are pruned with ADMM optimization tool (Deng et al, 2021), gradient-based rewiring (Chen et al, 2021), and lottery ticket hypothesis (Kim et al, 2022b). Meanwhile, various quantization techniques also have been proposed to compress SNNs (Datta et al, 2022;Guo et al, 2022b;Li et al, 2022a;Meng et al, 2022).…”

Section: Compression Methods For E Cient Snnsmentioning

confidence: 99%

See 1 more Smart Citation

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks

Kim,

Li,

Moitra

et al. 2023

Front. Neurosci.

Self Cite

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have gained increasing attention as energy-efficient neural networks owing to their binary and asynchronous computation. However, their non-linear activation, that is Leaky-Integrate-and-Fire (LIF) neuron, requires additional memory to store a membrane voltage to capture the temporal dynamics of spikes. Although the required memory cost for LIF neurons significantly increases as the input dimension goes larger, a technique to reduce memory for LIF neurons has not been explored so far. To address this, we propose a simple and effective solution, EfficientLIF-Net, which shares the LIF neurons across different layers and channels. Our EfficientLIF-Net achieves comparable accuracy with the standard SNNs while bringing up to ~4.3× forward memory efficiency and ~21.9× backward memory efficiency for LIF neurons. We conduct experiments on various datasets including CIFAR10, CIFAR100, TinyImageNet, ImageNet-100, and N-Caltech101. Furthermore, we show that our approach also offers advantages on Human Activity Recognition (HAR) datasets, which heavily rely on temporal information. The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/EfficientLIF-Net.

show abstract

Section: Compression Methods For E Cient Snnsmentioning

confidence: 99%

“…As pruning for SNNs is popular due to its usage on edge devices (Neftci et al, 2016 ; Shi et al, 2019 ; Guo et al, 2020 ; Chen et al, 2021 ; Kim et al, 2022b ), it is important to figure out whether the advantage from EfficientLIF-Net remains in sparse SNNs.…”

Section: Methodsmentioning

confidence: 99%

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks

Kim,

Li,

Moitra

et al. 2023

Front. Neurosci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The FI quantitatively measures the amount of information retained in a statistical model after being trained on a given data distribution [49]. Many prior works have used this metric to measure different aspects of deep learning models including SNN models [50,51]. Unlike prior works, we use pseudo-labels to generate FI instead of ground-truth labels.…”

Section: Evaluation Metrics 521 Fisher Information (Fi)mentioning

confidence: 99%

“…where, Tr(F) is the trace of FIM, ∇ is the partial derivative operator. We follow the same implementation as the algorithm specified in [51].…”

Section: Evaluation Metrics 521 Fisher Information (Fi)mentioning

confidence: 99%

Deep unsupervised learning using spike-timing-dependent plasticity

Lu,

Sengupta

2024

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

Spike-Timing-Dependent Plasticity (STDP) is an unsupervised learning mechanism for Spiking Neural Networks (SNNs) that has received significant attention from the neuromorphic hardware community. However, scaling such local learning techniques to deeper networks and large-scale tasks has remained elusive. In this work, we investigate a Deep-STDP framework where a rate-based convolutional network, that can be deployed in a neuromorphic setting, is trained in tandem with pseudo-labels generated by the STDP clustering process on the network outputs. We achieve $24.56\%$ higher accuracy and $3.5\times$ faster convergence speed at iso-accuracy on a 10-class subset of the Tiny ImageNet dataset in contrast to a $k$-means clustering approach.

show abstract

“…These winning tickets can be unearthed through iterative or one-shot pruning methods, where parameters with the smallest magnitudes are systematically removed [25]. Since its inception, LTH has garnered significant attention and been extensively studied [12,[25][26][27]. Zhang et al [12] not only employed dynamical systems theory and inertial manifold theory to theoretically substantiate the efficacy of the lottery ticket hypothesis, but also proposed a practical, lossless pruning solution.…”

Section: Model Pruningmentioning

confidence: 99%

Personalized Federated Learning Incorporating Adaptive Model Pruning at the Edge

Zhou,

Duan,

Qiu

et al. 2024

Electronics

View full text Add to dashboard Cite

Edge devices employing federated learning encounter several obstacles, including (1) the non-independent and identically distributed (Non-IID) nature of client data, (2) limitations due to communication bottlenecks, and (3) constraints on computational resources. To surmount the Non-IID data challenge, personalized federated learning has been introduced, which involves training tailored networks at the edge; nevertheless, these methods often exhibit inconsistency in performance. In response to these concerns, a novel framework for personalized federated learning that incorporates adaptive pruning of edge-side data is proposed in this paper. This approach, through a two-staged pruning process, creates customized models while ensuring strong generalization capabilities. Concurrently, by utilizing sparse models, it significantly condenses the model parameters, markedly diminishing both the computational burden and communication overhead on edge nodes. This method achieves a remarkable compression ratio of 3.7% on the Non-IID dataset FEMNIST, with the training accuracy remaining nearly unaffected. Furthermore, the total training duration is reduced by 46.4% when compared with the standard baseline method.

show abstract

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Cited by 18 publications

References 40 publications

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks

Deep unsupervised learning using spike-timing-dependent plasticity

Personalized Federated Learning Incorporating Adaptive Model Pruning at the Edge

Contact Info

Product

Resources

About