Forward Warping-Based Video Frame Interpolation Using a Motion Selective Network

Abstract

Recently, deep neural networks have shown surprising results in solving most of the traditional image processing problems. However, the video frame interpolation field does not show relatively good performance because the receptive field requires a vast spatio-temporal range. To reduce the computational complexity, in most frame interpolation studies, motion is first calculated with the optical flow, then interpolated frames are generated through backward warping. However, while the backward warping process is simple to implement, the interpolated image contains mixed motion and ghosting defects. Therefore, we propose a new network that does not use the backward warping method through the proposed max-min warping. Since max-min warping generates a clear warping image in advance according to the size of the motion and the network is configured to select the warping result according to the warped layer, using the proposed method, it is possible to optimize the computational complexity while selecting a contextually appropriate image. The video interpolation method using the proposed method showed 34.847 PSNR in the Vimeo90k dataset and 0.13 PSNR improvement compared to the Quadratic Video Interpolation method, showing that it is an efficient frame interpolation self-supervised learning.

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

“…Existing mainstream methods of video frame interpolation can be generally categorized into flow-based methods [7][8][9]11,12,[15][16][17][18][19][20][21][22][23][24][25][26], kernel-based methods [1,5,10,[27][28][29][30], phase-based methods [31], and hallucination-based methods [2,3,14,32].…”

“…SoftSplat [16] leverages the Softmax operator to mix pixels and features during the forward warping operation. Heo and Jeong [26] combine max-min warping with forward warping. ABME [18] further estimates an asym-metric bilateral optical flow based on symmetric optical flow.…”