Event-Driven Video Frame Synthesis

Abstract

Temporal Video Frame Synthesis (TVFS) aims at synthesizing novel frames at timestamps different from existing frames, which has wide applications in video codec, editing and analysis. In this paper, we propose a high framerate TVFS framework which takes hybrid input data from a lowspeed frame-based sensor and a high-speed event-based sensor. Compared to frame-based sensors, event-based sensors report brightness changes at very high speed, which may well provide useful spatio-temoral information for high framerate TVFS. In our framework, we first introduce a differentiable forward model to approximate the physical sensing process, fusing the two different modes of data as well as unifying a variety of TVFS tasks, i.e., interpolation, prediction and motion deblur. We leverage autodifferentiation which propagates the gradients of a loss defined on the measured data back to the latent high framerate video. We show results with better performance compared to state-ofthe-art. Second, we develop a deep learning-based strategy to enhance the results from the first step, which we refer as a residual "denoising" process. Our trained "denoiser" is beyond Gaussian denoising and shows properties such as contrast enhancement and motion awareness. We show that our framework is capable of handling challenging scenes including both fast motion and strong occlusions. Supplementary material, demo and code are released at: https://github.com/winswang/int-event-fusion/tree/win10.

An Asynchronous Kalman Filter for Hybrid Event Cameras

“…Pan et al [10], [28] devised the event double integral (EDI) relation between events and a blurry image, along with an optimisation approach to estimate contrast thresholds to reconstruct high-speed de-blurred video from events and frames. Reconstructed video can also be obtained by warping still images according to motion computed via events [47], [48], or by letting a neural network learn how to combine frames and events [15], [32], [49], [50], [51], [52]. Recognising the limited spatial resolution and noise of the DAVIS, some researchers built custom-built systems with separate eventframe sensors.…”

Removing Blocking Artifacts in Video Streams Using Event Cameras

“…While we believe we are the first to try image restoration in the context of removing blocking artifacts, there have been many publications that address aspects of the problem we face. Wang et al [1] for example used non-distorted video feeds along with events to interpolate video frames. Baura et al [20] and Rebecq et al [21] were early adopters of using learning methods to reconstruct intensity images from only events.…”

“…These benefits open up new paths in solving various vision problems. Event cameras have brought new solutions to many classical as well as novel problems in computer vision and robotics, including high frame-rate video reconstruction [1], [2], [3], with HDR [4], [5] and high resolution [6], [7], [8], and 3D reconstruction of human motion [9] and scenes [10], [11], as well as odometry [12], [13] and tracking [14], [15].…”

Frequency-Aware SVD Decomposition and its Application to Color Magnification and Motion Denoising

“…Analysis of video components along the time dimension is the typical approach to segregating temporally consistent and nonconsistent objects [20], [21]. Numerous approaches to enhancing the quality of video sequences have been proposed, including editing [22], synthesizing [23], [24], resampling [25], [26], and removing certain events and popping artifacts. Sunkavalli et al [27] introduced an editing approach that separates video components into their reflectance, illumination, and geometry factors.…”