A Fast 4K Video Frame Interpolation Using a Multi-Scale Optical Flow Reconstruction Network

Abstract

Recently, video frame interpolation research developed with a convolutional neural network has shown remarkable results. However, these methods demand huge amounts of memory and run time for high-resolution videos, and are unable to process a 4K frame in a single pass. In this paper, we propose a fast 4K video frame interpolation method, based upon a multi-scale optical flow reconstruction scheme. The proposed method predicts low resolution bi-directional optical flow, and reconstructs it into high resolution. We also proposed consistency and multi-scale smoothness loss to enhance the quality of the predicted optical flow. Furthermore, we use adversarial loss to make the interpolated frame more seamless and natural. We demonstrated that the proposed method outperforms the existing state-of-the-art methods in quantitative evaluation, while it runs up to 4.39× faster than those methods for 4K videos.

Video frame interpolation via optical flow estimation with image inpainting

“…According to different modeling methods, existing video frame interpolation methods can be roughly divided into pixel-based, 8,9 shape-based, 10,11 registration-based, 6,12,13 and learning-based. 14,15 As an early video interpolation strategy, the pixel-based method directly uses the gray information of two neighboring frames to construct a set of primary functions, for representing the inter-frame relationship. Similar to image interpolation technology, nearest-neighbor interpolation, linear interpolation, cubic B-spline function interpolation, 16 and other interpolation algorithms are often extended and used to obtain the new inter-frame images.…”

“…Inspired by optical flow methods, most of them represent the estimation of optical flow as a convolution process. 15,22 Furthermore, convolutional neural networks are directly used to hallucinate RGB values of video frames. [23][24][25] Owing to the powerful feature extraction capabilities of deep networks, the inter-frame images are represented by a learnable parameterized model rather than manually designed transformation kernels.…”

“…The key to solving frame interpolation is how to accurately model the motion of two neighboring frames and clearly reconstruct the inter‐frame pixels. According to different modeling methods, existing video frame interpolation methods can be roughly divided into pixel‐based, 8,9 shape‐based, 10,11 registration‐based, 6,12,13 and learning‐based 14,15 . As an early video interpolation strategy, the pixel‐based method directly uses the gray information of two neighboring frames to construct a set of primary functions, for representing the inter‐frame relationship.…”

“…With the development of deep learning, many video frame interpolation methods based on learning have been proposed. Inspired by optical flow methods, most of them represent the estimation of optical flow as a convolution process 15,22 . Furthermore, convolutional neural networks are directly used to hallucinate RGB values of video frames 23–25 .…”

Bivariate Thiele-Like Rational Interpolation Continued Fractions with Parameters Based on Virtual Points

“…The interpolation method also was applied to graphic image morphing and image processing [16][17][18][19][20][21][22][23][24]. In the paper [16], an effective directional Bayer color filter array demosaicking method based on residual interpolation is presented.…”