SpikeMS: Deep Spiking Neural Network for Motion Segmentation

Parameshwara, Chethan M.; Li, Simin; Fermüller, Cornelia; Sanket, Nitin J.; Evanusa, Matthew; Aloimonos, Yiannis

doi:10.1109/iros51168.2021.9636506

Cited by 20 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So far, SNNs have been used for classification tasks like image recognition [13], [29], object detection [30], [31], or motion segmentation [32]. Only a few works employed them for regression tasks.…”

Section: Related Workmentioning

confidence: 99%

StereoSpike: Depth Learning With a Spiking Neural Network

Rançon¹,

Cuadrado-Anibarro²,

Cottereau³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using StereoSpike, an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a modified U-Net-like encoder-decoder architecture. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction -the depth of each pixel-from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to near state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking neural network. Finally, we show that very low firing rates (<5%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.

show abstract

Section: Related Workmentioning

confidence: 99%

StereoSpike: Depth Learning With a Spiking Neural Network

Rançon¹,

Cuadrado-Anibarro²,

Cottereau³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…SNNs trained by the surrogate gradient method achieve high performance on complex datasets such as the Canadian Institute for Advanced Research (CIFAR) dataset (54), the Dynamic Vision Sensor (DVS) Gesture dataset (55) and the challenging Im-ageNet dataset (19) using only a few simulation time steps (56)(57)(58)(59)(60)(61), while SNNs converted from ANNs attain almost the same accuracy as that of the original ANNs on the ImageNet dataset with dozens of simulation time steps (51,62,63). Because of the rapid progress achieved by deep learning methods, the applications of SNNs have been expanded beyond classification to other tasks including object detection (64)(65)(66), object segmentation (67,68), depth estimation (69), and optical flow estimation (70). The boom exhibited by the research community indicates that spiking deep learning has become a promising research hotspot.…”

Section: Emerging Spiking Deep Learning Methodsmentioning

confidence: 99%

SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence

Fang,

Chen,

Ding

et al. 2023

Sci. Adv.

View full text Add to dashboard Cite

Spiking neural networks (SNNs) aim to realize brain-inspired intelligence on neuromorphic chips with high energy efficiency by introducing neural dynamics and spike properties. As the emerging spiking deep learning paradigm attracts increasing interest, traditional programming frameworks cannot meet the demands of the automatic differentiation, parallel computation acceleration, and high integration of processing neuromorphic datasets and deployment. In this work, we present the SpikingJelly framework to address the aforementioned dilemma. We contribute a full-stack toolkit for preprocessing neuromorphic datasets, building deep SNNs, optimizing their parameters, and deploying SNNs on neuromorphic chips. Compared to existing methods, the training of deep SNNs can be accelerated 11×, and the superior extensibility and flexibility of SpikingJelly enable users to accelerate custom models at low costs through multilevel inheritance and semiautomatic code generation. SpikingJelly paves the way for synthesizing truly energy-efficient SNN-based machine intelligence systems, which will enrich the ecology of neuromorphic computing.

show abstract

“…Our network architecture is similar to that in SpikeMS [ 32 ]. The end-to-end spike neural network model includes four spike feature coding layers and four spike feature decoding layers, as illustrated in Figure 5 .…”

Section: Methodsmentioning

confidence: 99%

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

Zhang

Zhao

et al. 2023

Micromachines

View full text Add to dashboard Cite

The advantages of an event camera, such as low power consumption, large dynamic range, and low data redundancy, enable it to shine in extreme environments where traditional image sensors are not competent, especially in high-speed moving target capture and extreme lighting conditions. Optical flow reflects the target’s movement information, and the target’s detailed movement can be obtained using the event camera’s optical flow information. However, the existing neural network methods for optical flow prediction of event cameras has the problems of extensive computation and high energy consumption in hardware implementation. The spike neural network has spatiotemporal coding characteristics, so it can be compatible with the spatiotemporal data of an event camera. Moreover, the sparse coding characteristic of the spike neural network makes it run with ultra-low power consumption on neuromorphic hardware. However, because of the algorithmic and training complexity, the spike neural network has not been applied in the prediction of the optical flow for the event camera. For this case, this paper proposes an end-to-end spike neural network to predict the optical flow of the discrete spatiotemporal data stream for the event camera. The network is trained with the spatio-temporal backpropagation method in a self-supervised way, which fully combines the spatiotemporal characteristics of the event camera while improving the network performance. Compared with the existing methods on the public dataset, the experimental results show that the method proposed in this paper is equivalent to the best existing methods in terms of optical flow prediction accuracy, and it can save 99% more power consumption than the existing algorithm, which is greatly beneficial to the hardware implementation of the event camera optical flow prediction., laying the groundwork for future low-power hardware implementation of optical flow prediction for event cameras.

show abstract

SpikeMS: Deep Spiking Neural Network for Motion Segmentation

Cited by 20 publications

References 32 publications

StereoSpike: Depth Learning With a Spiking Neural Network

StereoSpike: Depth Learning With a Spiking Neural Network

SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

Contact Info

Product

Resources

About