Multi-Cue Event Information Fusion for Pedestrian Detection With Neuromorphic Vision Sensors

Abstract

Neuromorphic vision sensors are bio-inspired cameras that naturally capture the dynamics of a scene with ultra-low latency, filtering out redundant information with low power consumption. Few works are addressing the object detection with this sensor. In this work, we propose to develop pedestrian detectors that unlock the potential of the event data by leveraging multi-cue information and different fusion strategies. To make the best out of the event data, we introduce three different event-stream encoding methods based on Frequency, Surface of Active Event (SAE) and Leaky Integrate-and-Fire (LIF). We further integrate them into the state-of-the-art neural network architectures with two fusion approaches: the channel-level fusion of the raw feature space and decision-level fusion with the probability assignments. We present a qualitative and quantitative explanation why different encoding methods are chosen to evaluate the pedestrian detection and which method performs the best. We demonstrate the advantages of the decision-level fusion via leveraging multi-cue event information and show that our approach performs well on a self-annotated event-based pedestrian dataset with 8,736 event frames. This work paves the way of more fascinating perception applications with neuromorphic vision sensors.

Efficient Spike-Driven Learning With Dendritic Event-Based Processing

“…Recently, neuromorphic computing has wide applications. Neuromorphic vision sensors capture the features of biological retina, which has changed the landscape of computer vision in both industry and academia (Chen et al, 2019;Zhou et al, 2019). Although neuromorphic systems with deep learning capability are still in research phases, the development of neuromorphic computing is calling for more biologically realistic processing strategies.…”

Neuromorphic Vision Datasets for Pedestrian Detection, Action Recognition, and Fall Detection

“…Standard computer vision algorithms cannot be used directly to process event data (Tan et al, 2015; Iyer et al, 2018). To address this problem, we introduce three encoding approaches here as Frequency (Chen, 2018), SAE (Surface of Active Events) (Mueggler et al, 2017b) and LIF, (Leaky Integrate-and-Fire) (Burkitt, 2006) to convert the asynchronous event stream to frames (Chen et al, 2019). The event data encoding procedure is shown in Figure 1D.…”

Making Sense of Neuromorphic Event Data for Human Action Recognition

“…The existing work on neuromorphic vision sensing in computer vision can be grouped into three themes: object detection [47]- [49], pedestrian detection [50], [51] and hand gesture recognition [33]- [35]. There is only a little work on exploring the neuromorphic data beyond object detection addressing highly semantic applications, such as, multi class action recognition, which still poses an important challenge.…”