Multi-Frame Pyramid Refinement Network for Video Frame Interpolation

Abstract

Video frame interpolation aims at synthesizing new video frames in-between existing frames to generate higher frame rate video. Current methods usually use two adjacent frames to generate intermediate frames, but sometimes fail to handle challenges like large motion, occlusion, and motion blur. This paper proposes a multi-frame pyramid refinement network to effectively use spatio-temporal information contained in multiple frames (more than two). There are three technical contributions in the proposed network. First, a special coarse-to-fine framework is proposed to refine optical flows in-between multiple frames with residual flows at each pyramid level. Therefore, large motion and occlusion can be effectively estimated. Second, a 3D U-net feature extractor is used to excavate spatio-temporal context and restore texture, which tend to disappear at course pyramid levels. Third, a multi-step perceptual loss is adopted to preserve more details in intermediate frame. It is worth mentioning that our approach can be easily extended to multi-frame interpolation. Our network is trained end-to-end using more than 80K collected frame groups (25 frames per group). Experimental results on several independent datasets show that our approach can effectively handle challenging cases, and perform consistently better than other state-of-the-art methods.INDEX TERMS Video frame interpolation, multiple frames, spatio-temporal information, optical flow, coarse-to-fine framework, deep learning.

Multi-Scale Attention Generative Adversarial Networks for Video Frame Interpolation

“…Most state-of-the-art optical flow models use deep learning [25], which suggest that CNNs can understand motion information between frames. In order to get better results, many researchers merge optical flow estimation and video interpolation frame in a single model [2], [6], [7], [26]- [28]. Liu et al [6] designs a deep network with a voxel flow layer to synthesize video frames by flowing pixel values from input video volume.…”

“…Niklaus and Liu [27] apply pixel-wise contextual information extracted by a pre-trained network to estimated bidirectional flow, and uses a frame synthesis network to produce the interpolated frame in a context-aware fashion. Zhang et al [7] uses a 3D U-Net feature extractor to excavate spatio-temporal context and rebuild texture, and a coarse-to-fine architecture to improve optical flows estimation. Li et al [28] proposes a lightweight network to estimate optical flow at feature level and introduce a new sobolev loss achieve better results.…”

“…Most existing video interpolation methods usually use convolutional neural networks (CNN). These methods can be divided into two categories: based on interpolation convolution kernels estimation [4], [5] and based on optical flow estimation [2], [6], [7]. The first technique combines motion estimation and pixel synthesis into a single process, which produces a convolution kernel for each pixel.…”

Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions

“…However, objects often follow complex, non-linear trajectories. To this end, researchers recently focused on leveraging information from more than two neighboring frames [7,8,26,50,53].…”

DRVI: Dual Refinement for Video Interpolation

“…Many recently published papers focus on addressing the motion analysis. For example, FI-NET [2] computes optical flow at feature level instead of image level to make the motion estimation more accurate; [3] learns the latent motion features instead of learning the optical flow as the motion feature; [4] and [5] learn from 4 input images instead of 2 images and add some techniques like long short term memory (LSTM) and Multi-Frame Pyramid Refinement to predict the motions.…”