Brain-inspired global-local learning incorporated with neuromorphic computing

Wu, Yujie; Zhao, Rong; Zhu, Jun; Chen, Feng; Xu, Mingkun; Li, Guoqi; Song, Sen; Deng, Lei; Wang, Guanrui; Zheng, Hao; Pei, Jing; Zhang, Youhui; Zhao, Mingguo; Shi, Luping

doi:10.1038/s41467-021-27653-2

Cited by 51 publications

(60 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They suffer from expensive computation during the training process on complex network architecture [45]. Choosing the appropriate surrogate gradient function is also extensively studied: [27] optimizes the surrogate gradient using the finite difference method, [46] uses meta learning to estimate the function of spiking neurons. PLIF [47] proposes to learn the parameters in LIF neurons via gradient descent.…”

Section: Direct Training Snnmentioning

confidence: 99%

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Li¹,

Deng²,

Dong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spiking Neural Network (SNN), originating from the neural behavior in biology, has been recognized as one of the nextgeneration neural networks. Conventionally, SNNs can be obtained by converting from pre-trained Artificial Neural Networks (ANNs) by replacing the non-linear activation with spiking neurons without changing the parameters. In this work, we argue that simply copying and pasting the weights of ANN to SNN inevitably results in activation mismatch, especially for ANNs that are trained with batch normalization (BN) layers. To tackle the activation mismatch issue, we first provide a theoretical analysis by decomposing local conversion error to clipping error and flooring error, and then quantitatively measure how this error propagates throughout the layers using the second-order analysis. Motivated by the theoretical results, we propose a set of layer-wise parameter calibration algorithms, which adjusts the parameters to minimize the activation mismatch. Extensive experiments for the proposed algorithms are performed on modern architectures and large-scale tasks including ImageNet classification and MS COCO detection. We demonstrate that our method can handle the SNN conversion with batch normalization layers and effectively preserve the high accuracy even in 32 time steps. For example, our calibration algorithms can increase up to 65% accuracy when converting VGG-16 with BN layers.

show abstract

Section: Direct Training Snnmentioning

confidence: 99%

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Li¹,

Deng²,

Dong³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…few-shot learning, energy efficient and explainable) but with limited flexibility. In the future, we will focus on solving the generalization issue in various technology paths: 1) designing an SNN based Network Architecture Search (NAS) mechanism which similar to the Auto ML (42); 2) introducing a reinforcement learning based agent to generate learning rules that equal to the human prior knowledge (43); 3) utilizing biological brain assembling theories to build a learning logic based architecture (28,25).…”

Section: Discussionmentioning

confidence: 99%

“…The developed system shows great one-shot learning behavior compared to the DL. Meanwhile, (25) proposed a spike-based hybrid plasticity model for solving few-shot learning, continual learning, and fault-tolerance learning problems, it combines both local plasticity and global supervise information for multi-task learning.…”

Section: Introductionmentioning

confidence: 99%

The Spike Gating Flow: A Hierarchical Structure Based Spiking Neural Network for Online Gesture Recognition

Zhen¹,

Wang²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Action recognition is an exciting research avenue for artificial intelligence since it may be a game changer in the emerging industrial fields such as robotic visions and automobiles. However, current deep learning faces major challenges for such applications because of the huge computational cost and the inefficient learning. Hence, we develop a novel brain-inspired Spiking Neural Network (SNN) based system titled Spiking Gating Flow (SGF) for online action learning. The developed system consists of multiple SGF units which assembled in a hierarchical manner. A single SGF unit involves three layers: a feature extraction layer, an event-driven layer and a histogram-based training layer. To demonstrate the developed system capabilities, we employ a standard Dynamic Vision Sensor (DVS) gesture classification as a benchmark. The results indicate that we can achieve 87.5% accuracy which is comparable with Deep Learning (DL), but at smaller training/inference data number ratio 1.5:1. And only a single training epoch is required during the learning process. Meanwhile, to the best of our knowledge, this is the highest accuracy among the non-backpropagation algorithm based SNNs. At last, we conclude the few-shot learning paradigm of the developed network: 1) a hierarchical structure-based network design involves human prior knowledge; 2) SNNs for content based global dynamic feature detection.

show abstract

“…Cortical neurons are sparsely connected via dynamical synapses that can be weakened or strengthened (Waters and Helmchen, 2006 ; Seeman et al, 2018 ) by some mechanisms, such as activity-dependent or retrograde signaling from other neurons. Understanding such network architectures and molecular mechanisms, and then implementing them in the artificial systems may lead to brain-like machines that are able to perform complex tasks (Faghihi and Moustafa, 2017 ; Li et al, 2019 ; Wu et al, 2022 ). Biological neurons are composed of dendrites that take up the signals from other neurons, the soma that is involved in information processing, and the axon that passes on the generated action potential (AP) into the terminal synapses of the axon ( Figure 1A ).…”

Section: Introductionmentioning

confidence: 99%

A Synaptic Pruning-Based Spiking Neural Network for Hand-Written Digits Classification

Faghihi

Alashwal

Moustafa

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

A spiking neural network model inspired by synaptic pruning is developed and trained to extract features of hand-written digits. The network is composed of three spiking neural layers and one output neuron whose firing rate is used for classification. The model detects and collects the geometric features of the images from the Modified National Institute of Standards and Technology database (MNIST). In this work, a novel learning rule is developed to train the network to detect features of different digit classes. For this purpose, randomly initialized synaptic weights between the first and second layers are updated using average firing rates of pre- and postsynaptic neurons. Then, using a neuroscience-inspired mechanism named, “synaptic pruning” and its predefined threshold values, some of the synapses are deleted. Hence, these sparse matrices named, “information channels” are constructed so that they show highly specific patterns for each digit class as connection matrices between the first and second layers. The “information channels” are used in the test phase to assign a digit class to each test image. In addition, the role of feed-back inhibition as well as the connectivity rates of the second and third neural layers are studied. Similar to the abilities of the humans to learn from small training trials, the developed spiking neural network needs a very small dataset for training, compared to the conventional deep learning methods that have shown a very good performance on the MNIST dataset. This work introduces a new class of brain-inspired spiking neural networks to extract the features of complex data images.

show abstract

Brain-inspired global-local learning incorporated with neuromorphic computing

Cited by 51 publications

References 46 publications

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

The Spike Gating Flow: A Hierarchical Structure Based Spiking Neural Network for Online Gesture Recognition

A Synaptic Pruning-Based Spiking Neural Network for Hand-Written Digits Classification

Contact Info

Product

Resources

About