ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Abstract

Video frame interpolation (VFI) is currently a very active research topic, with applications spanning computer vision, post production and video encoding. VFI can be extremely challenging, particularly in sequences containing large motions, occlusions or dynamic textures, where existing approaches fail to offer perceptually robust interpolation performance. In this context, we present a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture. ST-MFNet employs a new multi-scale multi-flow predictor to estimate many-to-one intermediate flows, which are combined with conventional one-to-one optical flows to capture both large and complex motions. In order to enhance interpolation performance for various textures, a 3D CNN is also employed to model the content dynamics over an extended temporal window. Moreover, ST-MFNet has been trained within an ST-GAN framework, which was originally developed for texture synthesis, with the aim of further improving perceptual interpolation quality. Our approach has been comprehensively evaluated -compared with fourteen state-of-the-art VFI algorithms -clearly demonstrating that ST-MFNet consistently outperforms these benchmarks on varied and representative test datasets, with significant gains up to 1.09dB in PSNR for cases including large motions and dynamic textures. Project page: https://danielism97.github.io/ST-MFNet.

A Subjective Quality Study for Video Frame Interpolation

“…Specifically, we computed four features for each sequence: spatial information (SI), temporal information (TI), motion vector (MV) and dynamic texture parameter (DTP). The latter two were included because motion magnitude and complexity have direct impact on VFI [3]. The calculation of SI, TI and DTP can be found in [19] and MV is described in [20].…”

“…The dropped frames were then reconstructed using five different VFI methods using their neighbouring frames. These include frame repeating, frame averaging (from two adjacent frames), DVF [1], [2] and ST-MFNet [3]. The first two were included because they have very low computational complexity and produce unique artifact types, juddering and blurring respectively.…”

“…Video frame interpolation (VFI) has attracted increasing attention in the research community over the past few years [1][2][3]. By generating non-existent frames between two consecutive original frames in a video with a relatively low frame rate [4], VFI up-converts the temporal resolution of a video, increasing the motion consistency and overall perceptual quality.…”

“…In contrast, kernel-based methods [8,9] learn adaptive local kernels to synthesise the output pixels. Several methods [3,10] have also been developed to combine flow-based warping and kernel-based synthesis to achieve improved interpolation performance.…”

Enhancing Deformable Convolution Based Video Frame Interpolation with Coarse-To-Fine 3d Cnn

“…It is noted that most of the existing DefConv-based methods use only two adjacent frames to interpolate the middle frame, whereas the use of a larger window can provide richer information on the spatio-temporal characteristics of the input signal [13,16]. Moreover, when predicting multi-flows under the DefConv-based VFI framework, only 2D convolutional neural networks (CNNs) have been employed, which cannot provide an explicit way of extracting temporal information from the content.…”