Spiking Neural Networks with Laterally-Inhibited Self-Recurrent Units

Zhang, Wenrui

doi:10.1109/ijcnn52387.2021.9533726

Cited by 5 publications

(7 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the recurrent network, negative weights mainly provide the function of inhibitory stimulation. Here we follow the settings in previous research (Zhang and Li, 2021b , a ) and adopt fixed negative weights. In experiments, fixed negative weights can reduce the optimization complexity without significant performance loss, while providing stable inhibitory connections.…”

Section: Methodsmentioning

confidence: 99%

Composing recurrent spiking neural networks using locally-recurrent motifs and risk-mitigating architectural optimization

Zhang,

Geng,

2024

Front. Neurosci.

View full text Add to dashboard Cite

In neural circuits, recurrent connectivity plays a crucial role in network function and stability. However, existing recurrent spiking neural networks (RSNNs) are often constructed by random connections without optimization. While RSNNs can produce rich dynamics that are critical for memory formation and learning, systemic architectural optimization of RSNNs is still an open challenge. We aim to enable systematic design of large RSNNs via a new scalable RSNN architecture and automated architectural optimization. We compose RSNNs based on a layer architecture called Sparsely-Connected Recurrent Motif Layer (SC-ML) that consists of multiple small recurrent motifs wired together by sparse lateral connections. The small size of the motifs and sparse inter-motif connectivity leads to an RSNN architecture scalable to large network sizes. We further propose a method called Hybrid Risk-Mitigating Architectural Search (HRMAS) to systematically optimize the topology of the proposed recurrent motifs and SC-ML layer architecture. HRMAS is an alternating two-step optimization process by which we mitigate the risk of network instability and performance degradation caused by architectural change by introducing a novel biologically-inspired “self-repairing” mechanism through intrinsic plasticity. The intrinsic plasticity is introduced to the second step of each HRMAS iteration and acts as unsupervised fast self-adaptation to structural and synaptic weight modifications introduced by the first step during the RSNN architectural “evolution.” We demonstrate that the proposed automatic architecture optimization leads to significant performance gains over existing manually designed RSNNs: we achieve 96.44% on TI46-Alpha, 94.66% on N-TIDIGITS, 90.28% on DVS-Gesture, and 98.72% on N-MNIST. To the best of the authors' knowledge, this is the first work to perform systematic architecture optimization on RSNNs.

show abstract

Section: Methodsmentioning

confidence: 99%

Composing recurrent spiking neural networks using locally-recurrent motifs and risk-mitigating architectural optimization

Zhang,

Geng,

2024

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Connections. SNN models with recurrent connections were demonstrated to have good performance and larger application scope than SNNs only with forward connections, such as RSNN [28], ST-RSBP [12], RDS-BP [13], LISNN [14], LISR [15], ScSr-SNN [16], SC-ML [17], and SCRNN [29]. For example, in the LISNN model [14], lateral interaction is achieved through a trainable interaction kernel function, which is used to calculate the interaction weights between neighboring neurons.…”

Section: Snn With Recurrentmentioning

confidence: 99%

“…Nowadays, more research focuses on how to make spiking neural networks trained with the BP algorithm have recurrent connections like biological neurons. Some methods have been proposed to achieve spiking neural network models with recurrent connections, such as improved spiking neural networks with lateral interactions (LISNN) [14], laterally inhibited self-recurrent unit (LISR) [15], skip-connected self-recurrent SNN (ScSr-SNN) [16], and sparsely connected recurrent motif layer (SC-ML) [17]. Tese methods achieve recurrent connections within a layer or within a single neuron through fxed-weight intralayer connections, single-neuron self-recurrent connections, and sparsely connected intralayer neural connections.…”

Section: Introductionmentioning

confidence: 99%

Intralayer-Connected Spiking Neural Network with Hybrid Training Using Backpropagation and Probabilistic Spike-Timing Dependent Plasticity

Chen

Zhu

et al. 2023

International Journal of Intelligent Systems

View full text Add to dashboard Cite

Spiking neural networks (SNNs) are highly computationally efficient artificial intelligence methods due to their advantages in having a biologically plausible computational framework. Recent research has shown that SNN trained using backpropagation (SNN-BP) exhibits excellent performance and has shown great potential in tasks such as image classification and security detection. However, the backpropagation method limits the dynamics and biological plausibility of the neural models in SNN, which will limit the recognition and simulation performance of SNN. In order to make neural models more similar to biological neurons, this study proposes a leaky integrate-and-fire (LIF) neuron model with dense intralayer connections, as well as efficient forward and backward processes in BP training. The new model will make the interaction between neurons within the layer more frequent, enhancing the intrinsic information exchange capability of SNN. An effective probabilistic spike-timing dependent plasticity (STDP) method is also proposed to reduce the overweighted connections between neurons, as well as a hybrid training method using BP and probabilistic STDP. The training method combines the advantages of BP and STDP to improve the performance of SNN models. An intralayer-connected SNN with hybrid training (ISNN-HY) is proposed with the combination of these improvements. The proposed model was evaluated on three static image datasets and one neuromorphic dataset. The results showed that the performance of ISNN-HY is superior to that of other SNN-BP models. The proposed method also makes it possible to accurately simulate biological neural systems.

show abstract

“…In order to expand spiking neurons' temporal receptive fields, researchers have made various works according to specific tasks by referring to various ANN structures that already exist. In Zhang and Li ( 2021 ), the author added circular connections to SNN, but the weight of phantom connections was manually set. Zhang and Li ( 2019 ), the author also changed the network into a loop structure and proposed an effective training method.…”

Section: Introductionmentioning

confidence: 99%

Spiking neural network with working memory can integrate and rectify spatiotemporal features

Chen,

Liu,

Shi

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

In the real world, information is often correlated with each other in the time domain. Whether it can effectively make a decision according to the global information is the key indicator of information processing ability. Due to the discrete characteristics of spike trains and unique temporal dynamics, spiking neural networks (SNNs) show great potential in applications in ultra-low-power platforms and various temporal-related real-life tasks. However, the current SNNs can only focus on the information a short time before the current moment, its sensitivity in the time domain is limited. This problem affects the processing ability of SNN in different kinds of data, including static data and time-variant data, and reduces the application scenarios and scalability of SNN. In this work, we analyze the impact of such information loss and then integrate SNN with working memory inspired by recent neuroscience research. Specifically, we propose Spiking Neural Networks with Working Memory (SNNWM) to handle input spike trains segment by segment. On the one hand, this model can effectively increase SNN's ability to obtain global information. On the other hand, it can effectively reduce the information redundancy between adjacent time steps. Then, we provide simple methods to implement the proposed network architecture from the perspectives of biological plausibility and neuromorphic hardware friendly. Finally, we test the proposed method on static and sequential data sets, and the experimental results show that the proposed model can better process the whole spike train, and achieve state-of-the-art results in short time steps. This work investigates the contribution of introducing biologically inspired mechanisms, e.g., working memory, and multiple delayed synapses to SNNs, and provides a new perspective to design future SNNs.

show abstract

Spiking Neural Networks with Laterally-Inhibited Self-Recurrent Units

Cited by 5 publications

References 29 publications

Composing recurrent spiking neural networks using locally-recurrent motifs and risk-mitigating architectural optimization

Composing recurrent spiking neural networks using locally-recurrent motifs and risk-mitigating architectural optimization

Intralayer-Connected Spiking Neural Network with Hybrid Training Using Backpropagation and Probabilistic Spike-Timing Dependent Plasticity

Spiking neural network with working memory can integrate and rectify spatiotemporal features

Contact Info

Product

Resources

About