Space-Time Video Super-Resolution Using Temporal Profiles

Xiao, Zeyu; Xiong, Zhiwei; Fu, Xueyang; Liu, Dong; Zha, Zheng-Jun

doi:10.1145/3394171.3413667

Cited by 38 publications

(20 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[20] used optical flow to explicitly warp the features. [65] introduced a news perspective to exploit the STVSR problem from the temporal profile. Unlike previous methods, we propose a one-stage framework ZSM to directly learn the mapping between partial LR observations and HR video frames, achieving accurate and fast STVSR.…”

Section: Space-time Video Super-resolutionmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

174

210

View full text Add to dashboard Cite

In this paper, we address the space-time video super-resolution, which aims at generating a high-resolution (HR) slow-motion video from a low-resolution (LR) and low frame rate (LFR) video sequence. A naive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR). Nevertheless, temporal interpolation and spatial upscaling are intra-related in this problem. Two-stage approaches cannot fully make use of this natural property. Besides, state-of-the-art VFI or VSR deep networks usually have a large frame reconstruction module in order to obtain high-quality photo-realistic video frames, which makes the two-stage approaches have large models and thus be relatively time-consuming. To overcome the issues, we present a one-stage space-time video super-resolution framework, which can directly reconstruct an HR slow-motion video sequence from an input LR and LFR video. Instead of reconstructing missing LR intermediate frames as VFI models do, we temporally interpolate LR frame features of the missing LR frames capturing local temporal contexts by a feature temporal interpolation module. Extensive experiments on widely used benchmarks demonstrate that the proposed framework not only achieves better qualitative and quantitative performance on both clean and noisy LR frames but also is several times faster than recent state-of-the-art two-stage networks. The source code is released in https:// github.com/ Mukosame/ Zooming-Slow-Mo-CVPR-2020.

show abstract

Section: Space-time Video Super-resolutionmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

174

210

View full text Add to dashboard Cite

show abstract

“…Video frame interpolation (VFI) converts a low frame rate (LFR) video to a high frame rate (HFR) one between given consecutive input frames, thereby providing a visually better motion-smoothed video which is favorably perceived by human visual systems (HVS) [24,25]. Therefore, it is widely used for diverse applications, such as adaptive streaming [52], slow motion generation [2,18,28,30,37,44] and space-time super resolution [9,15,21,22,50,51,[53][54][55].…”

Section: Introductionmentioning

confidence: 99%

DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting

Oh¹,

Kim

2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose a novel joint deblurring and multi-frame interpolation (DeMFI) framework, called DeMFI-Net, which accurately converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate based on flow-guided attentive-correlation-based feature bolstering (FAC-FB) module and recursive boosting (RB), in terms of multi-frame interpolation (MFI). The DeMFI-Net jointly performs deblurring and MFI where its baseline version performs feature-flow-based warping with FAC-FB module to obtain a sharp-interpolated frame as well to deblur two center-input frames. Moreover, its extended version further improves the joint task performance based on pixel-flow-based warping with GRU-based RB. Our FAC-FB module effectively gathers the distributed blurry pixel information over blurry input frames in feature-domain to improve the overall joint performances, which is computationally efficient since its attentive correlation is only focused pointwise. As a result, our DeMFI-Net achieves stateof-the-art (SOTA) performances for diverse datasets with significant margins compared to the recent SOTA methods, for both deblurring and MFI. All source codes including pretrained DeMFI-Net are publicly available at https: //github.com/JihyongOh/DeMFI.

show abstract

“…A straightforward strategy to perform spatio-temporal video super-resolution (STVSR) is to cascade the VSR model and the TSR model to generate high-resolution high-frame rate (HR-HFR) video from lowresolution low-frame rate (LR-LFR) video. Nevertheless, this does not yield optimal results as it cannot fully utilize the available spatiotemporal information [67]. Recently, a few works [30,31,66,67], studied the problem of joint spatio-temporal video super-resolution.…”

Section: Introductionmentioning

confidence: 99%

“…Nevertheless, this does not yield optimal results as it cannot fully utilize the available spatiotemporal information [67]. Recently, a few works [30,31,66,67], studied the problem of joint spatio-temporal video super-resolution. Zooming Slow-Mo [66] proposed a one-stage STVSR framework using Deformable Convolutional LSTM.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ada-VSR: Adaptive Video Super-Resolution with Meta-Learning

Gupta

Jonnalagedda

Bhanu

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Most of the existing works in supervised spatio-temporal video super-resolution (STVSR) heavily rely on a large-scale external dataset consisting of paired low-resolution low-frame rate (LR-LFR) and high-resolution high-frame rate (HR-HFR) videos. Despite their remarkable performance, these methods make a prior assumption that the low-resolution video is obtained by down-scaling the highresolution video using a known degradation kernel, which does not hold in practical settings. Another problem with these methods is that they cannot exploit instance-specific internal information of a video at testing time. Recently, deep internal learning approaches have gained attention due to their ability to utilize the instancespecific statistics of a video. However, these methods have a large inference time as they require thousands of gradient updates to learn the intrinsic structure of the data. In this work, we present Adaptive Video Super-Resolution (Ada-VSR) which leverages external, as well as internal, information through meta-transfer learning and internal learning, respectively. Specifically, meta-learning is employed to obtain adaptive parameters, using a large-scale external dataset, that can adapt quickly to the novel condition (degradation model) of the given test video during the internal learning task, thereby exploiting external and internal information of a video for super-resolution. The model trained using our approach can quickly adapt to a specific video condition with only a few gradient updates, which reduces the inference time significantly. Extensive experiments on standard datasets demonstrate that our method performs favorably against various state-of-the-art approaches. The project page is available at https://agupt013.github.io/AdaVSR.html

show abstract

Space-Time Video Super-Resolution Using Temporal Profiles

Cited by 38 publications

References 30 publications

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting

Ada-VSR: Adaptive Video Super-Resolution with Meta-Learning

Contact Info

Product

Resources

About