Event-Based Frame Interpolation with Ad-hoc Deblurring

Abstract

The performance of video frame interpolation is inherently correlated with the ability to handle motion in the input scene. Even though previous works recognize the utility of asynchronous event information for this task, they ignore the fact that motion may or may not result in blur in the input video to be interpolated, depending on the length of the exposure time of the frames and the speed of the motion, and assume either that the input video is sharp, restricting themselves to frame interpolation, or that it is blurry, including an explicit, separate deblurring stage before interpolation in their pipeline. We instead propose a general method for event-based frame interpolation that performs deblurring ad-hoc and thus works both on sharp and blurry input videos. Our model consists in a bidirectional recurrent network that naturally incorporates the temporal dimension of interpolation and fuses information from the input frames and the events adaptively based on their temporal proximity. In addition, we introduce a novel real-world high-resolution dataset with events and color videos named HighREV, which provides a challenging evaluation setting for the examined task. Extensive experiments on the standard GoPro benchmark and on our dataset show that our network consistently outperforms previous state-of-the-art methods on frame interpolation, single image deblurring and the joint task of interpolation and deblurring. Our code and dataset will be made publicly available.

Improving Fast Auto-Focus with Event Polarity

“…Its unique asynchronous event output mechanism allows for a very high response time (order of µs), meanwhile greatly reducing the bandwidth requirements (compared to the image matrix output of frame-based cameras). Therefore, it has been widely used in object tracking [1,2], Simultaneous Localization and Mapping (SLAM) [3], visual scene image reconstruction [4], image motion deblurring [5,6], human pose estimation [7], and many other visual applications. Likewise, all the advantages of event cameras show their potential in fast AF under adverse conditions like dark, fast-moving, and high dynamic range scenes.…”