Real-time panoramic tracking for event cameras

Abstract

Event cameras are a paradigm shift in camera technology. Instead of full frames, the sensor captures a sparse set of events caused by intensity changes. Since only the changes are transferred, those cameras are able to capture quick movements of objects in the scene or of the camera itself. In this work we propose a novel method to perform camera tracking of event cameras in a panoramic setting with three degrees of freedom. We propose a direct camera tracking formulation, similar to state-of-the-art in visual odometry. We show that the minimal information needed for simultaneous tracking and mapping is the spatial position of events, without using the appearance of the imaged scene point. We verify the robustness to fast camera movements and dynamic objects in the scene on a recently proposed dataset [18] and self-recorded sequences.

Focus Is All You Need: Loss Functions for Event-Based Vision

“…Notice that it is not possible to use ρ = const since this leads to supp(I) = N e , which does not depend on the motion parameters θ we wish to optimize for. Using weighting functions with unit area (i.e., ∞ 0 ρ(λ)dλ = 1) allows us to interpret (25) as a convex combination of supports (24), thus setting the correct scale so that (25) has the same units as (24).…”

“…= ρ(λ)dλ is a primitive of ρ, and F (0) is constant. This is an advantageous expression compared to (25), since it states that supp(I) can be computed using the values of I(x) directly, without having to compute (24) for every threshold λ and then sum up the results. By using a continuous image formulation, we have analytically integrated the partial sums (24).…”

“…Event cameras, such as the Dynamic Vision Sensor (DVS) [1] posses outstanding properties compared to traditional cameras: very high dynamic range (140 dB vs. 60 dB), high temporal resolution (in the order of µs), and do not suffer from motion blur. Hence, event cameras have a large potential to tackle challenging scenarios for standard cameras (such as high speed and high dynamic range) in tracking [2][3][4][5][6][7][8][9], depth estimation [10][11][12][13][14][15][16][17][18][19], Simultaneous Localization and Mapping [20][21][22][23][24][25][26][27], and recognition [28][29][30][31][32], among other applications. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential.…”

Neuromorphic vision: From sensors to event‐based algorithms

“…The code provided in Source Code for Event Lifetime () augments each event with its lifetime, which is nothing but the time taken by moving brightness that caused the event to move one pixel. The code is based on the event lifetime estimation algorithm proposed in Mueggler et al (). Intensity image reconstruction: Codes developed in Source Code for Image Reconstruction () and Source Code for Image Reconstruction () reconstruct intensity images from the DVS as per the algorithms proposed in Kim et al () and Reinbacher et al (), respectively. Localization: Localization, also known as ego motion estimation algorithm is implemented in Source Code for Localization () inspired by the algorithm proposed in Reinbacher et al (). Object Recognition: Implementation of a spiking neural network to recognize objects from the DVS data (Orchard et al, ) is provided at Source Code for Object Recognition ().…”

“…The next stage of development in parallel localization and mapping was the introduction of 2D map estimation. The state-of-the-art works that generate 2D map while estimating 2D planar motion are Weikersdorfer et al (2013) and Reinbacher et al (2017a). The authors of Weikersdorfer and Conradt (2012) extended their work to 2D mapping and localization (Weikersdorfer et al, 2013).…”