Retinomorphic Object Detection in Asynchronous Visual Streams

Abstract

Due to high-speed motion blur and challenging illumination, conventional frame-based cameras have encountered an important challenge in object detection tasks. Neuromorphic cameras that output asynchronous visual streams instead of intensity frames, by taking the advantage of high temporal resolution and high dynamic range, have brought a new perspective to address the challenge. In this paper, we propose a novel problem setting, retinomorphic object detection, which is the first trial that integrates foveal-like and peripheral-like visual streams. Technically, we first build a large-scale multimodal neuromorphic object detection dataset (i.e., PKU-Vidar-DVS) over 215.5k spatio-temporal synchronized labels. Then, we design temporal aggregation representations to preserve the spatio-temporal information from asynchronous visual streams. Finally, we present a novel bio-inspired unifying framework to fuse two sensing modalities via a dynamic interaction mechanism. Our experimental evaluation shows that our approach has significant improvements over the state-of-the-art methods with the single-modality, especially in high-speed motion and low-light scenarios. We hope that our work will attract further research into this newly identified, yet crucial research direction. Our dataset can be available at https://www.pkuml.org/resources/pku-vidar-dvs.html.

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

“…Besides getting images, various tasks have been developed. Zhu et al (Zhu et al 2022b) and Li et al (Li et al 2022) propose object detection methods based on spike streams. SCFlow (Hu et al 2022) estimates optical flow directly from the binary spike streams based on a pyramidal network.…”

Learning Temporal-Ordered Representation for Spike Streams Based on Discrete Wavelet Transforms

“…Secondly, solving various downstream visual tasks from spikes. Hu et al (2022a) take the first step on spike-based optical flow estimation, and Li et al (2022a) explore object detection based on spikes.…”

“…Many researchers have begun to develop computer vision algorithms suitable for the spike camera, including image reconstruction (Zhu et al 2020(Zhu et al , 2021Zheng et al 2021;Nie et al 2020;Zhao, Xiong, and Huang 2020;Zhao et al 2021Zhao et al , 2022aShe and Qing 2022), denoising (Xu et al 2020;Chen et al 2022), detection (Li et al 2022a), tracking and recognition of high-speed moving objects (Huang et al 2022;Zhao et al 2022b), depth estimation and optical flow estimation (Hu et al 2022a). Despite these, due to the dense time sequence information and discrete data of the spike camera, it is not easy to directly apply the existing algorithms of traditional cameras to the spike camera.…”

Theories and methods for large-scale brain-inspired neural networks

“…其中 [102] 提出了异 步时空记忆网络的检测方法, 包括时域自适应采样模 块和循环卷积模块, 将时域连续的脉冲流自适应地划 分为离散的脉冲块并实现时域上的记忆状态传递, 提 高了对运动检测的鲁棒性和对时空特性的利用. 之后, Li等人 [103] 结合DVS和Vidar, 利用两种传感器的互补 性, 实现了极端环境下的仿视网膜的目标检测. 等人 [104] 构建了基于ANN-SNN的深度脉冲YOLO网络 模型, 在检测任务逼近深度网络模型同时显著性地降 低功耗.…”