Progressive Motion Context Refine Network for Efficient Video Frame Interpolation

Abstract

Recently, flow-based frame interpolation methods have achieved great success by first modeling optical flow between target and input frames, and then building synthesis network for target frame generation. However, above cascaded architecture can lead to large model size and inference delay, hindering them from mobile and real-time applications. To solve this problem, we propose a novel Progressive Motion Context Refine Network (PMCRNet) to predict motion fields and image context jointly for higher efficiency. Different from others that directly synthesize target frame from deep feature, we explore to simplify frame interpolation task by borrowing existing texture from adjacent input frames, which means that decoder in each pyramid level of our PMCRNet only needs to update easier intermediate optical flow, occlusion merge mask and image residual. Moreover, we introduce a new annealed multi-scale reconstruction loss to better guide the learning process of this efficient PMCRNet. Experiments on multiple benchmarks show that proposed approaches not only achieve favorable quantitative and qualitative results but also reduces current model size and running time significantly.

Cuboid-Net: A Multi-Branch Convolutional Neural Network for Joint Space-Time Video Super Resolution

“…Lee et al [30] proposed a method called "adaptive collaboration of flows (AdaCoF)" that can generate a target pixel by referring to a variable number of pixels from any position. Kong et al [31] introduced a Progressive Motion Context Refine Network for efficient frame interpolation. The network predicts motion fields and image context jointly, simplifying the task by reusing existing textures from adjacent input frames.…”