IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation

Kong, Lingtong; Jiang, Boyuan; Luo, Donghao; Chu, Wenqing; Huang, Xiaoming; Tai, Ying; Wang, Chengjie; Yang, Jie

doi:10.1109/cvpr52688.2022.00201

Cited by 100 publications

(56 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paradigm, referred to as flow-based VFI, was further developed in the learning-based VFI literature [24]. These methods adopted various techniques to enhance interpolation quality, including the use of contextual information [25], designing bespoke flow estimation module [1,7,8,24,[26][27][28][29], employing a coarse-to-fine refinement strategy [30][31][32][33], developing new warping operations [9,34,35] and adopting higher-order motion modelling with additional input frames [5,10,36]. Some researchers argue that the imposition of a oneto-one mapping between the target and source pixels can limit the ability of flow-based methods to handle complex motions.…”

Section: A Video Frame Interpolationmentioning

confidence: 99%

BVI-VFI: A Video Quality Database for Video Frame Interpolation

Danier¹,

Zhang²,

Bull³

2022

Preprint

View full text Add to dashboard Cite

Section: A Video Frame Interpolationmentioning

confidence: 99%

BVI-VFI: A Video Quality Database for Video Frame Interpolation

Danier¹,

Zhang²,

Bull³

2022

Preprint

View full text Add to dashboard Cite

“…Above objective can already supervise frame interpolation, however, it often drops into local minimum due to misaligned texture in large motion cases. To solve this problem, DAIN [6], SoftSplat [11] use additional pre-trained optical flow network, RIFE [17], IFRNet [32] leverage knowledge distillation from the teacher flow network. However, cascaded architecture and knowledge distillation can result in large inference delay and complicate training process.…”

Section: Loss Functionmentioning

confidence: 99%

Progressive Motion Context Refine Network for Efficient Video Frame Interpolation

Kong

Liu

Yang

2022

IEEE Signal Process. Lett.

Self Cite

View full text Add to dashboard Cite

Recently, flow-based frame interpolation methods have achieved great success by first modeling optical flow between target and input frames, and then building synthesis network for target frame generation. However, above cascaded architecture can lead to large model size and inference delay, hindering them from mobile and real-time applications. To solve this problem, we propose a novel Progressive Motion Context Refine Network (PMCRNet) to predict motion fields and image context jointly for higher efficiency. Different from others that directly synthesize target frame from deep feature, we explore to simplify frame interpolation task by borrowing existing texture from adjacent input frames, which means that decoder in each pyramid level of our PMCRNet only needs to update easier intermediate optical flow, occlusion merge mask and image residual. Moreover, we introduce a new annealed multi-scale reconstruction loss to better guide the learning process of this efficient PMCRNet. Experiments on multiple benchmarks show that proposed approaches not only achieve favorable quantitative and qualitative results but also reduces current model size and running time significantly.

show abstract

“…[23] estimates the flow by sampling the 3D spatio-temporal neighborhood of every output pixel. [27], [28], [47] refine the estimation of intermediate flows in order to capture large and complex motions. [3], [24], [25], [26], [30] utilize bidirectional flows to warp frames and additional modules to address occlusion.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

CDFI: Compression-Driven Network Design for Frame Interpolation

Ding

Liang

Zhu

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

DNN-based frame interpolation, which generates intermediate frames from two consecutive frames, is often dependent on model architectures with a large number of features, preventing their deployment on systems with limited resources, such as mobile devices. We present a compression-driven network design for frame interpolation that leverages model pruning through sparsity-inducing optimization to greatly reduce the model size while attaining higher performance. Concretely, we begin by compressing the recently proposed AdaCoF model and demonstrating that a 10× compressed AdaCoF performs similarly to its original counterpart, where different strategies for using layerwise sparsity information as a guide are comprehensively investigated under a variety of hyperparameter settings. We then enhance this compressed model by introducing a multi-resolution warping module, which improves visual consistency with multi-level details. As a result, we achieve a considerable performance gain with a quarter of the size of the original AdaCoF. In addition, our model performs favorably against other state-of-the-art approaches on a wide variety of datasets. We note that the suggested compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithms. The source code is available at https://github.com/tding1/CDFI.

show abstract

IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation

Cited by 100 publications

References 32 publications

BVI-VFI: A Video Quality Database for Video Frame Interpolation

BVI-VFI: A Video Quality Database for Video Frame Interpolation

Progressive Motion Context Refine Network for Efficient Video Frame Interpolation

CDFI: Compression-Driven Network Design for Frame Interpolation

Contact Info

Product

Resources

About