Neuromorphic vision: From sensors to event‐based algorithms

Abstract

Regardless of the marvels brought by the conventional frame‐based cameras, they have significant drawbacks due to their redundancy in data and temporal latency. This causes problem in applications where low‐latency transmission and high‐speed processing are mandatory. Proceeding along this line of thought, the neurobiological principles of the biological retina have been adapted to accomplish data sparsity and high dynamic range at the pixel level. These bio‐inspired neuromorphic vision sensors alleviate the more serious bottleneck of data redundancy by responding to changes in illumination rather than to illumination itself. This paper reviews in brief one such representative of neuromorphic sensors, the activity‐driven event‐based vision sensor, which mimics human eyes. Spatio‐temporal encoding of event data permits incorporation of time correlation in addition to spatial correlation in vision processing, which enables more robustness. Henceforth, the conventional vision algorithms have to be reformulated to adapt to this new generation vision sensor data. It involves design of algorithms for sparse, asynchronous, and accurately timed information. Theories and new researches have begun emerging recently in the domain of event‐based vision. The necessity to compile the vision research carried out in this sensor domain has turned out to be considerably more essential. Towards this, this paper reviews the state‐of‐the‐art event‐based vision algorithms by categorizing them into three major vision applications, object detection/recognition, object tracking, localization and mapping. This article is categorized under: Technologies > Machine Learning

Event Visualization and Trajectory Tracking of the Load Carried by Rotary Crane

“…Unfortunately, the frame-based vision has some disadvantages, e.g., high data redundancy, high bandwidth demand in short-latency usecases, or limited dynamic range [41]. The DVS (see Figure 10), sometimes called "silicon retina" [41,42], functions differently. Each pixel of the sensor operates separately and elicits its events immediately when pixel illuminance changes [43].…”

Event-based Extraction of Navigation Features from Unsupervised Learning of Optic Flow Patterns

“…Over the last decade, an increasing number of studies have used event-based data for computer vision, with performances sometimes better than those obtained from more classical frame-based cameras in applications like object recognition (Neil and Liu, 2016;Stromatias et al, 2017), or visual odometry (Gallego and Scaramuzza, 2017;Nguyen et al, 2019). These studies were all based on deep convolutional neural networks or SNNs, coupled with supervised learning or classification approaches (see (Lakshmi et al, 2019)). For example, (Zhu et al, 2019) used an artificial neural network (ANN) to predict the optic flow from event-based data collected from a camera mounted on the top of a car moving within an urban environment (see also (Zhu et al, 2018)).…”

Event Camera Based Real-Time Detection and Tracking of Indoor Ground Robots

“…Event camera-based algorithms for single or multiple object detection, pose estimation, and tracking (MOT) can be classified into three categories: feature-based, artificial neural network-based, and time surface-based [35]. Studies focusing on robot pose estimation using event cameras have been reported in the literature [59]- [61].…”

“…The event camera used in this study was affixed to a stationary mount on the ceiling to provide a fixed frame of reference. When an event camera moves, the background suffers from clutter, making it difficult to distinguish the object of interest [35].…”