Real-time, high-speed video decompression using a frame- and event-based DAVIS sensor

Brändli, Christian; Müller, Lorenz; Delbrück, Tobi

doi:10.1109/iscas.2014.6865228

Cited by 89 publications

(86 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, C is in reality neither constant nor uniform across the image plane. Rather, it strongly varies depending on factors such as the sign of the brightness change [12], the event rate (because of limited pixel bandwidth) [20], and the temperature [21]. Consequently, events cannot by directly integrated to recover accurate intensity images in practice.…”

Section: Video Reconstructionmentioning

confidence: 99%

High Speed and High Dynamic Range Video with an Event Camera

Rebecq

Ranftl

Koltun

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

355

339

View full text Add to dashboard Cite

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.

show abstract

Section: Video Reconstructionmentioning

confidence: 99%

High Speed and High Dynamic Range Video with an Event Camera

Rebecq

Ranftl

Koltun

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

355

339

View full text Add to dashboard Cite

show abstract

“…An animated version can be found here: https://youtu.be/LauQ6LWTkxM. [Matsuda et al, 2015], optical flow estimation , Rueckauer and Delbruck, 2016, Bardow et al, 2016, high dynamic range (HDR) image reconstruction [Cook et al, 2011, Reinbacher et al, 2016, mosaicing [Kim et al, 2014] and video compression [Brandli et al, 2014a]. In ego-motion estimation, event cameras have been used for pose tracking [Weikersdorfer and Conradt, 2012, Mueggler et al, 2014, and visual odometry and Simultaneous Localization and Mapping (SLAM) [Weikersdorfer et al, 2013, Censi and Scaramuzza, 2014, Kueng et al, 2016, Kim et al, 2016.…”

Section: Event Cameras and Applicationsmentioning

confidence: 99%

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

et al. 2017

View full text Add to dashboard Cite

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the output is composed of a sequence of asynchronous events rather than actual intensity images, traditional vision algorithms cannot be applied, so that a paradigm shift is needed. We introduce the problem of eventbased multi-view stereo (EMVS) for event cameras and propose a solution to it. Unlike traditional MVS methods, which address the problem of estimating dense 3D structure from a set of known viewpoints, EMVS estimates semi-dense 3D structure from an event camera with known trajectory. Our EMVS solution elegantly exploits two inherent properties of an event camera: (1) its ability to respond to scene edges-which naturally provide semi-dense geometric information without any pre-processing operationand (2) the fact that it provides continuous measurements as the sensor moves. Despite its simplicity (it can be implemented in a few lines of code), our algorithm is able to produce accurate, semi-dense depth maps, without requiring any explicit data association or intensity estimation. We successfully validate our method on both synthetic and real data. Our method is computationally very efficient and runs in real-time on a CPU. Abstract Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the output is composed of a sequence of asynchronous events rather than actual intensity images, traditional vision algorithms cannot be applied, so that a paradigm shift is needed. We introduce the problem of Event-based Multi-View Stereo (EMVS) for event cameras and propose a solution to it. Unlike traditional MVS methods, which address the problem of estimating dense 3D structure from a set of known viewpoints, EMVS estimates semidense 3D structure from an event camera with known trajectory. Our EMVS solution elegantly exploits two inherent properties of an event camera: (i) its ability to respond to scene edges-which naturally provide semi-dense geometric information without any preprocessing operation-and (ii) the fact that it provides continuous measurements as the sensor moves. Despite its simplicity (it can be implemented in a few lines of code), our algorithm is able to produce accurate, semidense depth maps, without requiring any explicit data association or intensity estimation. We successfully validate our method on both synthetic and real data. Our method is computationally very efficient and runs in real-time on a CPU.

show abstract

“…where δ (t) is a Dirac-delta function and δ p p p i (p p p) is a Kronecker delta function with indices associated with the pixel coordinates of p p p i and p p p. That is δ p p p i (p p p) = 1 when p p p = p p p i and zero otherwise. In this paper we use the common assumption that the contrast threshold c is constant [11], [18], [19], although, in practice it does vary somewhat with intensity, event-rate and other factors [14]. The integral of events is…”

Section: A Mathematical Representation and Notationmentioning

confidence: 99%

“…It is possible to compute the direct integral (6) using a similar approach to the direct integration schemes of [11], [14]. The drawback of this approach is integration of sensor noise, which results in drift, and undermines low temporal-frequency components of the estimate L K (p p p,t) over time.…”

Section: Continuous-time Filter For Convolved Eventsmentioning

confidence: 99%

Asynchronous Spatial Image Convolutions for Event Cameras

Scheerlinck

Barnes

Mahony

2019

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Spatial convolution is arguably the most fundamental of 2D image processing operations. Conventional spatial image convolution can only be applied to a conventional image, that is, an array of pixel values (or similar image representation) that are associated with a single instant in time. Event cameras have serial, asynchronous output with no natural notion of an image frame, and each event arrives with a different timestamp. In this paper, we propose a method to compute the convolution of a linear spatial kernel with the output of an event camera. The approach operates on the event stream output of the camera directly without synthesising pseudo-image frames as is common in the literature. The key idea is the introduction of an internal state that directly encodes the convolved image information, which is updated asynchronously as each event arrives from the camera. The state can be read-off as-often-as and whenever required for use in higher level vision algorithms for real-time robotic systems. We demonstrate the application of our method to corner detection, providing an implementation of a Harris corner-response "state" that can be used in real-time for feature detection and tracking on robotic systems.

show abstract

Real-time, high-speed video decompression using a frame- and event-based DAVIS sensor

Cited by 89 publications

References 9 publications

High Speed and High Dynamic Range Video with an Event Camera

High Speed and High Dynamic Range Video with an Event Camera

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

Asynchronous Spatial Image Convolutions for Event Cameras

Contact Info

Product

Resources

About