Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation

Abstract

Event cameras or neuromorphic cameras mimic the human perception system as they measure the per-pixel intensity change rather than the actual intensity level. In contrast to traditional cameras, such cameras capture new information about the scene at MHz frequency in the form of sparse events. The high temporal resolution comes at the cost of losing the familiar per-pixel intensity information. In this work we propose a variational model that accurately models the behaviour of event cameras, enabling reconstruction of intensity images with arbitrary frame rate in real-time. Our method is formulated on a per-event-basis, where we explicitly incorporate information about the asynchronous nature of events via an event manifold induced by the relative timestamps of events. In our experiments we verify that solving the variational model on the manifold produces high-quality images without explicitly estimating optical flow.

High Speed and High Dynamic Range Video with an Event Camera

“…This figure shows image reconstructions from each method, 0.5 seconds after the sensor was started. HF [5] and MR [4], which are based on event integration, cannot recover the intensity correctly, resulting in "edge" images (first and second row) or severe "ghosting" effects (third row, where the trace of the dartboard is clearly visible). In contrast, our network successfully reconstructs most of the scene accurately, even with a low number of events.…”

“…19. Qualitative comparison of our reconstruction method with two recent competing approaches, MR [4] and HF [5], on sequences from [38], which contain ground truth frames from a DAVIS240C sensor. Our method successfully reconstructs fine details (textures in the second and third row) compared to other methods, while avoiding ghosting effects (particulary visible in the shapes sequences on the fourth row).…”

“…Color reconstruction from such events was first shown by [44], where a single color image was recovered from a large set of events using a method similar to [17]. Later, [45] adapted existing monochrome video reconstruction methods [4], [30] to color (a) Individual channels (b) [45] (upsampled, color) (c) [45] (details) (d) Grayscale (full res) (e) Ours (full res, color) (f) Ours (details) Fig. 10.…”

“…Naive integration of the event stream leads to very fast degradation of image quality due to accumulating noise. As a remedy, earlier works have proposed hand-crafted image priors to constrain the problem [2], [3], [4], arXiv:1906.07165v1 [cs.CV] 15 Jun 2019 [5]. However, these priors make strong assumptions about the statistics of natural images, leading to unrealistic reconstructions and artifacts.…”

CED: Color Event Camera Dataset

“…Since their introduction, event cameras have spawned a flurry of research. They have been used in feature detection and tracking [3][4][5][6], depth estimation [7][8][9][10], stereo [11][12][13][14], optical flow [15][16][17][18], image reconstruction [19][20][21][22][23][24][25], localization [26][27][28][29], SLAM [30][31][32], visualinertial odometry [33][34][35][36], pattern recognition [37][38][39][40], and more. In response to the growing needs of the community, several important event-based vision datasets have been released, directed at popular topics such as SLAM [28], optical flow [41,42] and recognition [37,43].…”