Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation

Reinbacher, Christian; Graber, Gottfried; Pock, Thomas

doi:10.5244/c.30.9

Cited by 51 publications

(72 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An animated version can be found here: https://youtu.be/LauQ6LWTkxM. [Matsuda et al, 2015], optical flow estimation , Rueckauer and Delbruck, 2016, Bardow et al, 2016, high dynamic range (HDR) image reconstruction [Cook et al, 2011, Reinbacher et al, 2016, mosaicing [Kim et al, 2014] and video compression [Brandli et al, 2014a]. In ego-motion estimation, event cameras have been used for pose tracking [Weikersdorfer and Conradt, 2012, Mueggler et al, 2014, and visual odometry and Simultaneous Localization and Mapping (SLAM) [Weikersdorfer et al, 2013, Censi and Scaramuzza, 2014, Kueng et al, 2016, Kim et al, 2016.…”

Section: Event Cameras and Applicationsmentioning

confidence: 99%

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

et al. 2017

View full text Add to dashboard Cite

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the output is composed of a sequence of asynchronous events rather than actual intensity images, traditional vision algorithms cannot be applied, so that a paradigm shift is needed. We introduce the problem of eventbased multi-view stereo (EMVS) for event cameras and propose a solution to it. Unlike traditional MVS methods, which address the problem of estimating dense 3D structure from a set of known viewpoints, EMVS estimates semi-dense 3D structure from an event camera with known trajectory. Our EMVS solution elegantly exploits two inherent properties of an event camera: (1) its ability to respond to scene edges-which naturally provide semi-dense geometric information without any pre-processing operationand (2) the fact that it provides continuous measurements as the sensor moves. Despite its simplicity (it can be implemented in a few lines of code), our algorithm is able to produce accurate, semi-dense depth maps, without requiring any explicit data association or intensity estimation. We successfully validate our method on both synthetic and real data. Our method is computationally very efficient and runs in real-time on a CPU. Abstract Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the output is composed of a sequence of asynchronous events rather than actual intensity images, traditional vision algorithms cannot be applied, so that a paradigm shift is needed. We introduce the problem of Event-based Multi-View Stereo (EMVS) for event cameras and propose a solution to it. Unlike traditional MVS methods, which address the problem of estimating dense 3D structure from a set of known viewpoints, EMVS estimates semidense 3D structure from an event camera with known trajectory. Our EMVS solution elegantly exploits two inherent properties of an event camera: (i) its ability to respond to scene edges-which naturally provide semi-dense geometric information without any preprocessing operation-and (ii) the fact that it provides continuous measurements as the sensor moves. Despite its simplicity (it can be implemented in a few lines of code), our algorithm is able to produce accurate, semidense depth maps, without requiring any explicit data association or intensity estimation. We successfully validate our method on both synthetic and real data. Our method is computationally very efficient and runs in real-time on a CPU.

show abstract

Section: Event Cameras and Applicationsmentioning

confidence: 99%

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Sharpness Memory HDR Low light (a) DAVIS frame (b) MR [22] (c) HF [24] (d) E2VID [50] Figure 7: Edge cases for different reconstruction methods. First row: initialization, all method but E2VID fail.…”

Section: Fast Motionmentioning

confidence: 99%

“…Since their introduction, event cameras have spawned a flurry of research. They have been used in feature detection and tracking [3][4][5][6], depth estimation [7][8][9][10], stereo [11][12][13][14], optical flow [15][16][17][18], image reconstruction [19][20][21][22][23][24][25], localization [26][27][28][29], SLAM [30][31][32], visualinertial odometry [33][34][35][36], pattern recognition [37][38][39][40], and more. In response to the growing needs of the community, several important event-based vision datasets have been released, directed at popular topics such as SLAM [28], optical flow [41,42] and recognition [37,43].…”

Section: Introductionmentioning

confidence: 99%

CED: Color Event Camera Dataset

Scheerlinck

Rebecq

Stoffregen

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Event cameras are novel, bio-inspired visual sensors, whose pixels output asynchronous and independent timestamped spikes at local intensity changes, called 'events'. Event cameras offer advantages over conventional framebased cameras in terms of latency, high dynamic range (HDR) and temporal resolution. Until recently, event cameras have been limited to outputting events in the intensity channel, however, recent advances have resulted in the development of color event cameras, such as the Color-DAVIS346. In this work, we present and release the first Color Event Camera Dataset (CED), containing 50 minutes of footage with both color frames and events. CED features a wide variety of indoor and outdoor scenes, which we hope will help drive forward event-based vision research. We also present an extension of the event camera simulator ESIM [1] that enables simulation of color events. Finally, we present an evaluation of three state-of-the-art image reconstruction methods that can be used to convert the Color-DAVIS346 into a continuous-time, HDR, color video camera to visualise the event stream, and for use in downstream vision applications.

show abstract

“…where δ (t) is a Dirac-delta function and δ p p p i (p p p) is a Kronecker delta function with indices associated with the pixel coordinates of p p p i and p p p. That is δ p p p i (p p p) = 1 when p p p = p p p i and zero otherwise. In this paper we use the common assumption that the contrast threshold c is constant [11], [18], [19], although, in practice it does vary somewhat with intensity, event-rate and other factors [14]. The integral of events is…”

Section: A Mathematical Representation and Notationmentioning

confidence: 99%

Asynchronous Spatial Image Convolutions for Event Cameras

Scheerlinck

Barnes

Mahony

2019

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Spatial convolution is arguably the most fundamental of 2D image processing operations. Conventional spatial image convolution can only be applied to a conventional image, that is, an array of pixel values (or similar image representation) that are associated with a single instant in time. Event cameras have serial, asynchronous output with no natural notion of an image frame, and each event arrives with a different timestamp. In this paper, we propose a method to compute the convolution of a linear spatial kernel with the output of an event camera. The approach operates on the event stream output of the camera directly without synthesising pseudo-image frames as is common in the literature. The key idea is the introduction of an internal state that directly encodes the convolved image information, which is updated asynchronously as each event arrives from the camera. The state can be read-off as-often-as and whenever required for use in higher level vision algorithms for real-time robotic systems. We demonstrate the application of our method to corner detection, providing an implementation of a Harris corner-response "state" that can be used in real-time for feature detection and tracking on robotic systems.

show abstract

Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation

Cited by 51 publications

References 14 publications

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

CED: Color Event Camera Dataset

Asynchronous Spatial Image Convolutions for Event Cameras

Contact Info

Product

Resources

About