Asynchronous, Photometric Feature Tracking Using Events and Frames

Gehrig, Daniel; Rebecq, Henri; Gallego, Guillermo; Scaramuzza, Davide

doi:10.1007/978-3-030-01258-8_46

Cited by 113 publications

(77 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early approaches did not reconstruct videos, but focused on the reconstruction of a single image from a large set of events collected by an event camera moving through a static scene. These works exploit the fact that every event provides one equation relating the intensity gradient and optic flow through brightness constancy [15]. Cook et al [10] used bio-inspired, interconnected networks to simultaneously recover intensity images, optic flow, and angular velocity from an event camera performing small rotations.…”

Section: Related Workmentioning

confidence: 99%

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Rebecq

Ranftl

Koltun

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

276

227

View full text Add to dashboard Cite

Event cameras are novel sensors that report brightness changes in the form of asynchronous "events" instead of intensity frames. They have significant advantages over conventional cameras: high temporal resolution, high dynamic range, and no motion blur. Since the output of event cameras is fundamentally different from conventional cameras, it is commonly accepted that they require the development of specialized algorithms to accommodate the particular nature of events. In this work, we take a different view and propose to apply existing, mature computer vision techniques to videos reconstructed from event data. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. Our experiments show that our approach surpasses state-of-the-art reconstruction methods by a large margin (> 20%) in terms of image quality. We further apply off-the-shelf computer vision algorithms to videos reconstructed from event data on tasks such as object classification and visual-inertial odometry, and show that this strategy consistently outperforms algorithms that were specifically designed for event data. We believe that our approach opens the door to bringing the outstanding properties of event cameras to an entirely new range of tasks. A video of the experiments is available at https://youtu.be/IdYrC4cUO0I

show abstract

Section: Related Workmentioning

confidence: 99%

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Rebecq

Ranftl

Koltun

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

276

227

View full text Add to dashboard Cite

show abstract

“…Event cameras such as the DAVIS and DVS [3,17] report log intensity changes, inspired by human vision. Although several works try to explore the advantages of the high temporal resolution provided by event cameras [39,13,26,41,40,8,15], how to make the best use of the event camera has not yet been fully investigated.…”

Section: Related Workmentioning

confidence: 99%

Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera

Pan

Scheerlinck

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

178

145

View full text Add to dashboard Cite

Event-based cameras can measure intensity changes (called 'events') with microsecond accuracy under highspeed motion and challenging lighting conditions. With the active pixel sensor (APS), the event camera allows simultaneous output of the intensity frames. However, the output images are captured at a relatively low frame-rate and often suffer from motion blur. A blurry image can be regarded as the integral of a sequence of latent images, while the events indicate the changes between the latent images. Therefore, we are able to model the blur-generation process by associating event data to a latent image. In this paper, we propose a simple and effective approach, the Event-based Double Integral (EDI) model, to reconstruct a high framerate, sharp video from a single blurry frame and its event data. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Experimental results on both synthetic and real images demonstrate the superiority of our EDI model and optimization method in comparison to the state-of-the-art.

show abstract

“…In contrast, methods operating on event packets trade-off latency for computational efficiency and performance. Despite their differences, both paradigms have been successfully applied on various vision tasks, including tracking [19,21,40,42], depth estimation [3,52,67], visual odometry [27,54,57,66], recognition [29,44], and optical flow estimation [7,69]. A good survey on the applications of event cameras can be found in [18].…”

Section: Introductionmentioning

confidence: 99%

End-to-End Learning of Representations for Asynchronous Event-Based Data

Gehrig

Loquercio

Derpanis

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

231

237

View full text Add to dashboard Cite

Event cameras are vision sensors that record asynchronous streams of per-pixel brightness changes, referred to as "events". They have appealing advantages over frame-based cameras for computer vision, including high temporal resolution, high dynamic range, and no motion blur. Due to the sparse, non-uniform spatiotemporal layout of the event signal, pattern recognition algorithms typically aggregate events into a grid-based representation and subsequently process it by a standard vision pipeline, e.g., Convolutional Neural Network (CNN). In this work, we introduce a general framework to convert event streams into grid-based representations through a sequence of differentiable operations. Our framework comes with two main advantages: (i) allows learning the input event representation together with the task dedicated network in an end-to-end manner, and (ii) lays out a taxonomy that unifies the majority of extant event representations in the literature and identifies novel ones. Empirically, we show that our approach to learning the event representation end-to-end yields an improvement of approximately 12% on optical flow estimation and object recognition over state-of-the-art methods.

show abstract

Asynchronous, Photometric Feature Tracking Using Events and Frames

Cited by 113 publications

References 36 publications

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera

End-to-End Learning of Representations for Asynchronous Event-Based Data

Contact Info

Product

Resources

About