Abstract:Event cameras are novel, bio-inspired visual sensors, whose pixels output asynchronous and independent timestamped spikes at local intensity changes, called 'events'. Event cameras offer advantages over conventional framebased cameras in terms of latency, high dynamic range (HDR) and temporal resolution. Until recently, event cameras have been limited to outputting events in the intensity channel, however, recent advances have resulted in the development of color event cameras, such as the Color-DAVIS346. In t… Show more
“…Figs. 21, 22 and 23 present HDR reconstruction results on various publicly available datasets [5], [43], [45]. Further results are shown in the supplementary video which conveys these results in a better form than still images.…”
Section: D1 Results On Synthetic Event Datamentioning
confidence: 94%
“…Indoors [45] Outdoors [5] Night Drive [43] Night Drive [5] (a) Events (b) Frame (c) Reconstruction Fig. 23.…”
Section: D2 Additional Qualitative Results On Real Datamentioning
confidence: 99%
“…Video reconstruction under challenging lighting. First row: indoor sequence [45]. Second row: outdoor sequence from [5].…”
Section: D2 Additional Qualitative Results On Real Datamentioning
confidence: 99%
“…Color reconstruction from such events was first shown by [44], where a single color image was recovered from a large set of events using a method similar to [17]. Later, [45] adapted existing monochrome video reconstruction methods [4], [30] to color (a) Individual channels (b) [45] (upsampled, color) (c) [45] (details) (d) Grayscale (full res) (e) Ours (full res, color) (f) Ours (details) Fig. 10.…”
Section: Color Video Reconstructionmentioning
confidence: 99%
“…We now describe a simple method to perform color reconstruction from color event data at full resolution with our network, and then present qualitative results. Following [45], we reconstruct the four color channels independently at quarter resolution ( Fig. 10(a)), upsample with bicubic interpolation, and recombine them into a low-quality color image ( Fig.…”
Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.
“…Figs. 21, 22 and 23 present HDR reconstruction results on various publicly available datasets [5], [43], [45]. Further results are shown in the supplementary video which conveys these results in a better form than still images.…”
Section: D1 Results On Synthetic Event Datamentioning
confidence: 94%
“…Indoors [45] Outdoors [5] Night Drive [43] Night Drive [5] (a) Events (b) Frame (c) Reconstruction Fig. 23.…”
Section: D2 Additional Qualitative Results On Real Datamentioning
confidence: 99%
“…Video reconstruction under challenging lighting. First row: indoor sequence [45]. Second row: outdoor sequence from [5].…”
Section: D2 Additional Qualitative Results On Real Datamentioning
confidence: 99%
“…Color reconstruction from such events was first shown by [44], where a single color image was recovered from a large set of events using a method similar to [17]. Later, [45] adapted existing monochrome video reconstruction methods [4], [30] to color (a) Individual channels (b) [45] (upsampled, color) (c) [45] (details) (d) Grayscale (full res) (e) Ours (full res, color) (f) Ours (details) Fig. 10.…”
Section: Color Video Reconstructionmentioning
confidence: 99%
“…We now describe a simple method to perform color reconstruction from color event data at full resolution with our network, and then present qualitative results. Following [45], we reconstruct the four color channels independently at quarter resolution ( Fig. 10(a)), upsample with bicubic interpolation, and recombine them into a low-quality color image ( Fig.…”
Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.
Event cameras are paradigm-shifting novel sensors that report asynchronous, per-pixel brightness changes called 'events' with unparalleled low latency. This makes them ideal for high speed, high dynamic range scenes where conventional cameras would fail. Recent work has demonstrated impressive results using Convolutional Neural Networks (CNNs) for video reconstruction and optic flow with events. We present strategies for improving training data for event based CNNs that result in 20-40% boost in performance of existing state-of-the-art (SOTA) video reconstruction networks retrained with our method, and up to 15% for optic flow networks. A challenge in evaluating event based video reconstruction is lack of quality ground truth images in existing datasets. To address this, we present a new High Quality Frames (HQF) dataset, containing events and ground truth frames from a DAVIS240C that are well-exposed and minimally motion-blurred. We evaluate our method on HQF + several existing major event camera datasets.Video, code and datasets: https://timostoff.github.io/20ecnn
This paper has been accepted for publication at the European Conference on Computer Vision, 2020Reducing the Sim-to-Real Gap for Event Cameras 3 that provides perfectly aligned frames from an integrated Active Pixel Sensor (APS). HQF also contains a diverse range of motions and scene types, including slow motion and pauses that are challenging for event based video reconstruction. We quantitatively evaluate our method on two major event camera datasets: IJRR [23] and MVSEC [42], in addition to our HQF, demonstrating gains of 20-40 % for video reconstruction and up to 15 % for optic flow when we retrain existing SOTA networks.Contribution We present a method to generate synthetic training data that improves generalizability to real event data, guided by statistical analysis of existing datasets. We additionally propose a simple method for dynamic train-time noise augmentation that yields up to 10 % improvement for video reconstruction. Using our method, we retrain several network architectures from previously published works on video reconstruction [28,32] and optic flow [43, 44] from events. We are able to show significant improvements that persist over architectures and tasks. Thus, we believe our findings will provide invaluable insight for others who wish to train models on synthetic events for a variety of tasks. We provide a new comprehensive High Quality Frames dataset targeting ground truth image frames for video reconstruction evaluation. Finally, we provide our data generation code, training set, training code and our pretrained models, together with dozens of useful helper scripts for the analysis of event-based datasets to make this task easier for fellow researchers.In summary, our major contributions are:-A method for simulating training data that yields 20 %-40 and up to 15 % improvement for event based video reconstruction and optic flow CNNs. -Dynamic train-time event noise augmentation.-A novel High Quality Frames dataset.-Extensive ...
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.