A New Spiking Convolutional Recurrent Neural Network (SCRNN) With Applications to Event-Based Hand Gesture Recognition

Abstract

The combination of neuromorphic visual sensors and spiking neural network offers a high efficient bio-inspired solution to real-world applications. However, processing event- based sequences remains challenging because of the nature of their asynchronism and sparsity behavior. In this paper, a novel spiking convolutional recurrent neural network (SCRNN) architecture that takes advantage of both convolution operation and recurrent connectivity to maintain the spatial and temporal relations from event-based sequence data are presented. The use of recurrent architecture enables the network to have a sampling window with an arbitrary length, allowing the network to exploit temporal correlations between event collections. Rather than standard ANN to SNN conversion techniques, the network utilizes a supervised Spike Layer Error Reassignment (SLAYER) training mechanism that allows the network to adapt to neuromorphic (event-based) data directly. The network structure is validated on the DVS gesture dataset and achieves a 10 class gesture recognition accuracy of 96.59% and an 11 class gesture recognition accuracy of 90.28%.

Hyperconverged autonomous organic reaction infrastructure (HAORI) driven by SpecSNN, for low dielectric constant polymer research

“…Another serious shortcoming of the RNN is its pseudo time dependence, which could not precisely reflect the intensity change rate. As a solution, inspired by Xing, 25 we use the spiking coded difference between spectra as the training dataset and replace the traditional RNN with the spiking neural network (SNN), as illustrated in Fig. 2B.…”

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

“…Most of the learning-based SNN work has so far been focused on problems like classification [10,57,59], optical estimation [35,15], motion segmentation [33], and angular velocity regression [11]. Moreover, Lee et al [23] proposed an ANN-SNN hybrid architecture for optical estimation, using SNN as encoder and ANN as decoder and residual block.…”

“…Visual systems [32,1] constructed with SNN and event cameras have demonstrated their capacity in solving visual tasks as well as prominent energy-efficiency. However, most of the SNN work has so far been focused on problems like classification [10,57,59], optical estimation [35,15], motion segmentation [33], and angular velocity regression [11]. To the best of our knowledge, we are the first to attempt event-based image reconstruction task based on a fully deep SNN architecture.…”

Mental State Assessment in College English Teaching Courses Based on Deep Learning

“…For example, emotional robots can perform human-computer interaction through both facial expression and speech [8][9][10]. erefore, the study of multimodal emotion recognition methods based on facial expressions and speech plays a very important role in the development and progress of future technologies such as emotional robots, unmanned driving, and intelligent transportation [11][12][13].…”