High Speed and High Dynamic Range Video with an Event Camera

Abstract

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.

Event-Based Vision: A Survey

“…In recent years, this technology has attracted a lot of attention from academia and industry. This is due to the availability of prototype event cameras and the advantages that they offer to tackle problems that are difficult with standard frame-based image sensors (that provide stroboscopic synchronous sequences of pictures), such as high-speed motion estimation [6], [7] or high dynamic range (HDR) imaging [8].…”

“…Event cameras are used for object tracking [12], [13], surveillance and monitoring [14], and object/gesture recognition [15], [16], [17]. They are also profitable for depth estimation [18], [19], structured light 3D scanning [20], optical flow estimation [21], [22], HDR image reconstruction [8], [23], [24] and Simultaneous Localization and Mapping (SLAM) [25], [26], [27]. Event-based vision is a growing field of research, and other applications, such as image deblurring [28] or star tracking [29], [30], will appear as event cameras become widely available [9].…”

High-speed particle detection and tracking in microfluidic devices using event-based sensing

“…This could be tackled by engineering new tools to help with the calibration either in the setup itself or computationally. 58 Finally, the drastic changes in datafrom images to events imposes the development of a new framework for processing. Importantly, it has been demonstrated here that data can be analysed to extract relevant information (e.g., particle focusing position, particle velocity) and can also be directly compared to images (e.g., comparison with composite image from fluorescent imaging).…”

Integrating Event Camera Sensor Emulator

“…The integrator accumulates incoming photons as photoelectrons, and the decimator uses a decimation factor 2 to divide the event frequency, determining the threshold of light the pixel must take in before firing an event [9]. Hence such a sensor is not invariant to scene illumination, but can directly encode scene luminosity without the need for conventional active pixels like DAVIS [3,5,10], estimation [7], or a complex neural network to reconstruct video [6]. The incident light intensity, , of a pixel may be computed by dividing the decimation factor by the time delta, Δ , between two consecutive events for that pixel, written as ∝ 2 Δ .…”