Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Lee, Chankyu; Kosta, Adarsh Kumar; Zhu, Alex Zihao; Chaney, Kenneth; Daniilidis, Kostas; Roy, Kaushik

doi:10.1007/978-3-030-58526-6_22

Cited by 118 publications

(128 citation statements)

References 28 publications

Supporting

Mentioning

123

Contrasting

Order By: Relevance

“…Methods above provide solutions with both advantages and disadvantages, and recent research makes optimization by combining complementary advantages that each category has to offer. Current innovation architectures tailored to high-rate, variablelength, and nonuniform event stream have been proposed to balance the trade-off between accuracy and efficiency while protecting the spatio-temporal sparsity [4,43,44]. Generally speaking, with high capacity neural networks, features are automatically learned from data by optimizing the corresponding object function in data-driven approaches, exempt from handcrafted feature descriptors.…”

Section: Data-driven Approachesmentioning

confidence: 99%

Data‐Driven Technology in Event‐Based Vision

Sun

Shi²,

Zhang³

et al. 2021

Complexity

View full text Add to dashboard Cite

Event cameras which transmit per-pixel intensity changes have emerged as a promising candidate in applications such as consumer electronics, industrial automation, and autonomous vehicles, owing to their efficiency and robustness. To maintain these inherent advantages, the trade-off between efficiency and accuracy stands as a priority in event-based algorithms. Thanks to the preponderance of deep learning techniques and the compatibility between bio-inspired spiking neural networks and event-based sensors, data-driven approaches have become a hot spot, which along with the dedicated hardware and datasets constitute an emerging field named event-based data-driven technology. Focusing on data-driven technology in event-based vision, this paper first explicates the operating principle, advantages, and intrinsic nature of event cameras, as well as background knowledge in event-based vision, presenting an overview of this research field. Then, we explain why event-based data-driven technology becomes a research focus, including reasons for the rise of event-based vision and the superiority of data-driven approaches over other event-based algorithms. Current status and future trends of event-based data-driven technology are presented successively in terms of hardware, datasets, and algorithms, providing guidance for future research. Generally, this paper reveals the great prospects of event-based data-driven technology and presents a comprehensive overview of this field, aiming at a more efficient and bio-inspired visual system to extract visual features from the external environment.

show abstract

Section: Data-driven Approachesmentioning

confidence: 99%

Data‐Driven Technology in Event‐Based Vision

Sun

Shi²,

Zhang³

et al. 2021

Complexity

View full text Add to dashboard Cite

show abstract

“…However, SNNs for processing temporal or spatiotemporal data are still primarily based on recurrent connections (DePasquale et al, 2016 ; Bellec et al, 2018 ) and use supervised training (Stromatias et al, 2017 ; Wu et al, 2018 ), leading to high network complexity for processing spatiotemporal data and high demand for labeled data. Recently, SNN has also been explored for optical flow applications, such as Spike-FlowNet (Lee et al, 2020 ), which is based on self-supervision; with STDP-based learning, it is possible to implement SNN for optical flow (Paredes-Vallés et al, 2020 ), but learning is limited to only short-term temporal patterns.…”

Section: Introductionmentioning

confidence: 99%

“…Our key innovation is to use feedforward connections of spiking neurons with different dynamics to represent memory of different time-scales and learn temporal patterns. This eliminates the need for recurrent connection present in state-of-the-art SNNs used for temporal learning (DePasquale et al, 2016 ; Bellec et al, 2018 ; Wu et al, 2018 ; Lee et al, 2020 ). Moreover, we adapt spiking convolution modules (Kheradpisheh et al, 2018 ) to the network architecture.…”

Section: Introductionmentioning

confidence: 99%

A Heterogeneous Spiking Neural Network for Unsupervised Learning of Spatiotemporal Patterns

et al. 2021

View full text Add to dashboard Cite

This paper introduces a heterogeneous spiking neural network (H-SNN) as a novel, feedforward SNN structure capable of learning complex spatiotemporal patterns with spike-timing-dependent plasticity (STDP) based unsupervised training. Within H-SNN, hierarchical spatial and temporal patterns are constructed with convolution connections and memory pathways containing spiking neurons with different dynamics. We demonstrate analytically the formation of long and short term memory in H-SNN and distinct response functions of memory pathways. In simulation, the network is tested on visual input of moving objects to simultaneously predict for object class and motion dynamics. Results show that H-SNN achieves prediction accuracy on similar or higher level than supervised deep neural networks (DNN). Compared to SNN trained with back-propagation, H-SNN effectively utilizes STDP to learn spatiotemporal patterns that have better generalizability to unknown motion and/or object classes encountered during inference. In addition, the improved performance is achieved with 6x fewer parameters than complex DNNs, showing H-SNN as an efficient approach for applications with constrained computation resources.

show abstract

“…Owing to their different advantages, in recent years there is a growing trend of integrating ANNs and SNNs to explore hybrid neural networks (HNNs) toward artificial general intelligence ( Marblestone et al, 2016 ; Zhang et al, 2016 ; Ullman, 2019 ). For example, in some cases of event-driven tasks ( Srinivasan and Roy, 2019 ; Lee et al, 2020 ), researchers use SNN modules for abstracting sparse temporal information, and further combine ANN modules for improving the classification performance. Similarly, in some cases of static image processing tasks ( Kheradpisheh et al, 2018 ; Chancán et al, 2020 ), researchers use ANN modules to extract the edge contrasts in images and further process them with SNN modules for low power consumption.…”

Section: Introductionmentioning

confidence: 99%

“…Hybrid neural networks have a promising perspective on the development of artificial general intelligence. However, by far these models are mainly studied and implemented on general-purpose platforms (i.e., CPU or GPU) ( Kheradpisheh et al, 2018 ; Srinivasan and Roy, 2019 ; Chancán et al, 2020 ; Lee et al, 2020 ). On the other side, HNNs retain the basic properties of neural networks, being promising in high-efficiency implementation on domain-specific hardware platforms ( Sze et al, 2017 ).…”

Section: Introductionmentioning

confidence: 99%

End-to-End Implementation of Various Hybrid Neural Networks on a Cross-Paradigm Neuromorphic Chip

Wang

et al. 2021

Front. Neurosci.

View full text Add to dashboard Cite

Integration of computer-science oriented artificial neural networks (ANNs) and neuroscience oriented spiking neural networks (SNNs) has emerged as a highly promising direction to achieve further breakthroughs in artificial intelligence through complementary advantages. This integration needs to support individual modeling of ANNs and SNNs as well as their hybrid modeling, which not only simultaneously calculates single-paradigm networks but also converts their different information representations. It remains challenging to realize effective calculation and signal conversion on the existing dedicated hardware platforms. To solve this problem, we propose an end-to-end mapping framework for implementing various hybrid neural networks on many-core neuromorphic architectures based on the cross-paradigm Tianjic chip. We construct hardware configuration schemes for four typical signal conversions and establish a global timing adjustment mechanism among different heterogeneous modules. Experimental results show that our framework can implement these hybrid models with low execution latency and low power consumption with nearly no accuracy degradation. This work provides a new approach of developing hybrid neural network models for brain-inspired computing chips and further tapping the potential of these models.

show abstract

Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Cited by 118 publications

References 28 publications

Data‐Driven Technology in Event‐Based Vision

Data‐Driven Technology in Event‐Based Vision

A Heterogeneous Spiking Neural Network for Unsupervised Learning of Spatiotemporal Patterns

End-to-End Implementation of Various Hybrid Neural Networks on a Cross-Paradigm Neuromorphic Chip

Contact Info

Product

Resources

About