Low-latency visual odometry using event-based feature tracks

Abstract

New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional camera and an event-based sensor in the same pixel array. These sensors have great potential for robotics because they allow us to combine the benefits of conventional cameras with those of eventbased sensors: low latency, high temporal resolution, and high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. In this paper, we present a lowlatency visual odometry algorithm for the DAVIS sensor using event-based feature tracks. Features are first detected in the grayscale frames and then tracked asynchronously using the stream of events. The features are then fed to an event-based visual odometry algorithm that tightly interleaves robust pose optimization and probabilistic mapping. We show that our method successfully tracks the 6-DOF motion of the sensor in natural scenes. This is the first work on event-based visual odometry with the DAVIS sensor using feature tracks. Low-Latency Visual Odometry using Event-based Feature Tracks Beat Kueng, Elias Mueggler, Guillermo Gallego and Davide ScaramuzzaAbstract-New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional camera and an event-based sensor in the same pixel array. These sensors have great potential for robotics because they allow us to combine the benefits of conventional cameras with those of event-based sensors: low latency, high temporal resolution, and high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. In this paper, we present a lowlatency visual odometry algorithm for the DAVIS sensor using event-based feature tracks. Features are first detected in the grayscale frames and then tracked asynchronously using the stream of events. The features are then fed to an event-based visual odometry algorithm that tightly interleaves robust pose optimization and probabilistic mapping. We show that our method successfully tracks the 6-DOF motion of the sensor in natural scenes. This is the first work on event-based visual odometry with the DAVIS sensor using feature tracks. MULTIMEDIA MATERIALA video attachment to this work is available on the authors' webpage.

EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time

“…A visual odometry system operating in a parallel trackingand-mapping manner was presented in [11]. The system recovered 6-DOF motions in natural scenes by tracking a sparse set of features using the event stream.…”

The Event-Driven Software Library for YARP—With Algorithms and iCub Applications

“…Detection of a moving ball by a moving robot was achieved at rates of over 500 Hz (Glover and Bartolozzi, 2016). Visual tracking of features was shown at a rate higher than standard cameras (Vasco et al, 2016a) and also features position could be updated "between frames" of a standard camera (Kueng et al, 2016).…”

Vertical landing for micro air vehicles using event‐based optical flow

“…For example, the method presented in Ref. uses frames to identify visual features and events to track their position in high‐speed motion in order to perform visual odometry. Early work on optical flow estimation using the DVS was shown in Ref.…”