BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation

Park, Junheum; Ko, Keunsoo; Lee, Chul

doi:10.48550/arxiv.2007.12622

Cited by 7 publications

(20 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One can see that their interpola-tions normally contain visible artifacts and are not capable of preserving clear details. Note that BMBC [45] occasionally synthesizes sharp results but is not as consistent as ours. We conjecture that the additional bilateral cost volume in BMBC benefits the intermediate motion estimations, which can also be incorporated into our design.…”

Section: Qualitative Evaluationcontrasting

confidence: 59%

“…[36] estimates the flow by sampling from the 3D spatio-temporal neighborhood of each output pixel. [27,45,62,63] utilize bi-directional flows to warp frames and resort to additional modules to handle occlusion. [41,42] integrate an off-the-shelf flow model [55] into the network.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

“…In contrast, our method built upon it predicts sharper and more realistic results due to our newly added FP module and PS mechanism. Furthermore, we compare with BMBC [45], CAIN [16] and EDSC [11], which are all newly developed within the year. In particular, similar to AdaCoF, EDSC relies on the deformable separable convolution but estimates an extra mask to help with the image synthesis.…”

Section: Qualitative Evaluationmentioning

confidence: 99%

“…The compressed AdaCoF even outperforms the full one in this case. rapid expansion in optical flow [1,60], many approaches either utilize an off-the-shelf flow model [2,41,42,61] or estimate their own task-specific flow [27,36,62,63,45] as a guidance of pixel-level motion interpolation. However, integrating a pre-trained flow model makes the whole architecture cumbersome, while with only pixel-level information the task-oriented flow alone is still insufficient in handling complex occlusion and blur.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

CDFI: Compression-Driven Network Design for Frame Interpolation

Ding¹,

Liang²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

DNN-based frame interpolation-that generates the intermediate frames given two consecutive frames-typically relies on heavy model architectures with a huge number of features, preventing them from being deployed on systems with limited resources, e.g., mobile devices. We propose a compression-driven network design for frame interpolation (CDFI), that leverages model pruning through sparsityinducing optimization to significantly reduce the model size while achieving superior performance. Concretely, we first compress the recently proposed AdaCoF model and show that a 10× compressed AdaCoF performs similarly as its original counterpart; then we further improve this compressed model by introducing a multi-resolution warping module, which boosts visual consistencies with multi-level details. As a consequence, we achieve a significant performance gain with only a quarter in size compared with the original AdaCoF. Moreover, our model performs favorably against other state-of-the-arts in a broad range of datasets. Finally, the proposed compression-driven framework is generic and can be easily transferred to other DNNbased frame interpolation algorithm. Our source code is available at https://github.com/tding1/CDFI. * Equal contribution. This work was done when Tianyu Ding was an intern at Applied Sciences Group, Microsoft.† Corresponding author.Recently, a large number of researches have been conducted in this area, especially those based on deep neural networks (DNN) for their promising results in motion esti-

show abstract

Section: Qualitative Evaluationcontrasting

confidence: 59%

Section: Video Frame Interpolationmentioning

confidence: 99%

Section: Qualitative Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

CDFI: Compression-Driven Network Design for Frame Interpolation

Ding¹,

Liang²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lee et al [24] proposed adaptive collaboration of flows as a new warping module to deal with complex motions. To handle motions more explicitly, many flow-based video interpolation approaches [1], [2], [18], [32], [38], [39], [43] have also been proposed. However, these methods usually have inherent issues with inaccuracies and missing information from optical flow results.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

174

210

View full text Add to dashboard Cite

In this paper, we address the space-time video super-resolution, which aims at generating a high-resolution (HR) slow-motion video from a low-resolution (LR) and low frame rate (LFR) video sequence. A naive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR). Nevertheless, temporal interpolation and spatial upscaling are intra-related in this problem. Two-stage approaches cannot fully make use of this natural property. Besides, state-of-the-art VFI or VSR deep networks usually have a large frame reconstruction module in order to obtain high-quality photo-realistic video frames, which makes the two-stage approaches have large models and thus be relatively time-consuming. To overcome the issues, we present a one-stage space-time video super-resolution framework, which can directly reconstruct an HR slow-motion video sequence from an input LR and LFR video. Instead of reconstructing missing LR intermediate frames as VFI models do, we temporally interpolate LR frame features of the missing LR frames capturing local temporal contexts by a feature temporal interpolation module. Extensive experiments on widely used benchmarks demonstrate that the proposed framework not only achieves better qualitative and quantitative performance on both clean and noisy LR frames but also is several times faster than recent state-of-the-art two-stage networks. The source code is released in https:// github.com/ Mukosame/ Zooming-Slow-Mo-CVPR-2020.

show abstract

CDFI: Compression-Driven Network Design for Frame Interpolation

Ding

Liang

Zhu

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

DNN-based frame interpolation, which generates intermediate frames from two consecutive frames, is often dependent on model architectures with a large number of features, preventing their deployment on systems with limited resources, such as mobile devices. We present a compression-driven network design for frame interpolation that leverages model pruning through sparsity-inducing optimization to greatly reduce the model size while attaining higher performance. Concretely, we begin by compressing the recently proposed AdaCoF model and demonstrating that a 10× compressed AdaCoF performs similarly to its original counterpart, where different strategies for using layerwise sparsity information as a guide are comprehensively investigated under a variety of hyperparameter settings. We then enhance this compressed model by introducing a multi-resolution warping module, which improves visual consistency with multi-level details. As a result, we achieve a considerable performance gain with a quarter of the size of the original AdaCoF. In addition, our model performs favorably against other state-of-the-art approaches on a wide variety of datasets. We note that the suggested compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithms. The source code is available at https://github.com/tding1/CDFI.

show abstract

BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation

Cited by 7 publications

References 33 publications

CDFI: Compression-Driven Network Design for Frame Interpolation

CDFI: Compression-Driven Network Design for Frame Interpolation

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

CDFI: Compression-Driven Network Design for Frame Interpolation

Contact Info

Product

Resources

About