A Flexible Recurrent Residual Pyramid Network for Video Frame Interpolation

Zhang, Haoxian; Zhao, Yang; Wang, Ronggang

doi:10.1007/978-3-030-58595-2_29

Cited by 37 publications

(20 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Partial cost volume is used to represent matching cost associated with different disparities. Inspired by classical pyramid energy minimization in optical flow algorithms, RRPN [30] designed a recurrent residual pyramid architecture for video frame interpolation to refine optical flow using a shared network for every pyramid level. Following above methods, we also exploit the advantages of classical principles of optical flow -the pyramid structure, multi-scale warping, and cost volume.…”

Section: B Pyramid Structure and The Cost Volumementioning

confidence: 99%

See 1 more Smart Citation

PDWN: Pyramid Deformable Warping Network for Video Interpolation

Chen

Wang

Liu³

et al. 2021

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

Video interpolation aims to generate a non-existent intermediate frame given the past and future frames. Many state-of-the-art methods achieve promising results by estimating the optical flow between the known frames and then generating the backward flows between the middle frame and the known frames. However, these methods usually suffer from the inaccuracy of estimated optical flows and require additional models or information to compensate for flow estimation errors. Following the recent development in using deformable convolution (DConv) for video interpolation, we propose a light but effective model, called Pyramid Deformable Warping Network (PDWN). PDWN uses a pyramid structure to generate DConv offsets of the unknown middle frame with respect to the known frames through coarse-to-fine successive refinements. Cost volumes between warped features are calculated at every pyramid level to help the offset inference. At the finest scale, the two warped frames are adaptively blended to generate the middle frame. Lastly, a context enhancement network further enhances the contextual detail of the final output. Ablation studies demonstrate the effectiveness of the coarse-to-fine offset refinement, cost volumes, and DConv. Our method achieves better or on-par accuracy compared to state-of-the-art models on multiple datasets while the number of model parameters and the inference time are substantially less than previous models. Moreover, we present an extension of the proposed framework to use four input frames, which can achieve significant improvement over using only two input frames, with only a slight increase in the model size and inference time.

show abstract

Section: B Pyramid Structure and The Cost Volumementioning

confidence: 99%

“…al. [30] designed a recurrent residual pyramid architecture to refine optical flow using a shared network across pyramid levels. Other methods, despite the usage of multi-scale features, only generate one-stage optical flow [3,8,16].…”

Section: Introductionmentioning

confidence: 99%

PDWN: Pyramid Deformable Warping Network for Video Interpolation

Chen

Wang

Liu³

et al. 2021

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

show abstract

“…Various efforts have been made to alleviate this issue, including the use of depth information [5], higher order motion models [33,60], and adaptive forward warping [41]. A second group of methods [25,34,44,45,61,65] have been developed to improve approximation by directly predicting intermediate flows. These approaches typically employ a coarse-to-fine architecture, which supports a larger receptive field for capturing large motions.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

“…While flowbased methods use the estimated optical flow maps to warp input frames, kernel-based methods learn local or shared convolution kernels for synthesizing the output. To handle challenging scenarios encountered in VFI applications, various techniques have been employed to enhance these methods, including non-linear motion models [45,50,60], coarse-to-fine architectures [10,44,50,65], attention mechanisms [12,29], and deformable convolutions [21,32].…”

Section: Introductionmentioning

confidence: 99%

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Danier¹,

Zhang²,

Bull³

2021

Preprint

View full text Add to dashboard Cite

Video frame interpolation (VFI) is currently a very active research topic, with applications spanning computer vision, post production and video encoding. VFI can be extremely challenging, particularly in sequences containing large motions, occlusions or dynamic textures, where existing approaches fail to offer perceptually robust interpolation performance. In this context, we present a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture. ST-MFNet employs a new multi-scale multi-flow predictor to estimate many-to-one intermediate flows, which are combined with conventional one-to-one optical flows to capture both large and complex motions. In order to enhance interpolation performance for various textures, a 3D CNN is also employed to model the content dynamics over an extended temporal window. Moreover, ST-MFNet has been trained within an ST-GAN framework, which was originally developed for texture synthesis, with the aim of further improving perceptual interpolation quality. Our approach has been comprehensively evaluated -compared with fourteen state-of-the-art VFI algorithms -clearly demonstrating that ST-MFNet consistently outperforms these benchmarks on varied and representative test datasets, with significant gains up to 1.09dB in PSNR for cases including large motions and dynamic textures. Project page: https://danielism97.github.io/ST-MFNet.

show abstract

“…However, VFI is significantly challenging, which is attributed to diverse factors such as occlusions, large motions and change of light. Recent deeplearning-based VFI has been actively studied, showing remarkable performances [48,4,7,37,25,13,31,51,6,33]. However, they are often optimized for existing LFR benchmark datasets of low resolution (LR), which may lead to poor VFI performance, especially for videos of 4K resolution (4096×2160) or higher with very large motion [1,21].…”

Section: Introductionmentioning

confidence: 99%

XVFI: eXtreme Video Frame Interpolation

Sim

Kim

2021

Preprint

View full text Add to dashboard Cite

a) 96.6 (b) 71.0 (c) 34.9 (d) 40.6 (e) 196.5 (f) 152.2 Figure 1. Some examples of our X4K1000FPS dataset, which contain diverse motions in 4K-resolution of 1000-fps. The numbers below the examples are the magnitude means of optical flows between two input frames in 30 fps. This is a video figure that can be best viewed with motion using Adobe™ Reader. It should be noted that they are rendered in down-scales at 15 fps for visualization convenience.

show abstract

A Flexible Recurrent Residual Pyramid Network for Video Frame Interpolation

Cited by 37 publications

References 34 publications

PDWN: Pyramid Deformable Warping Network for Video Interpolation

PDWN: Pyramid Deformable Warping Network for Video Interpolation

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

XVFI: eXtreme Video Frame Interpolation

Contact Info

Product

Resources

About