Space-Time Distillation for Video Super-Resolution

Xiao, Zeyu; Fu, Xueyang; Huang, Jie; Cheng, Zhen; Xiong, Zhiwei

doi:10.1109/cvpr46437.2021.00215

Cited by 38 publications

(7 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Lightening VSR Model with Knowledge Distillation. There is a considerable performance gap between the lightweight VSR model and the normally used complex VSR model, while the latter one requires a much larger amount of resources (Xiao et al 2021). This problem is particularly acute on resource-limited devices, e.g., smartphones and wearable devices.…”

Section: Discussionmentioning

confidence: 99%

Optical Flow for Video Super-Resolution: A Survey

Tu¹,

Li²,

Xie³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video super-resolution is currently one of the most active research topics in computer vision as it plays an important role in many visual applications. Generally, video super-resolution contains a significant component, i.e., motion compensation, which is used to estimate the displacement between successive video frames for temporal alignment. Optical flow, which can supply dense and sub-pixel motion between consecutive frames, is among the most common ways for this task. To obtain a good understanding of the effect that optical flow acts in video super-resolution, in this work, we conduct a comprehensive review on this subject for the first time. This investigation covers the following major topics: the function of super-resolution (i.e., why we require super-resolution); the concept of video super-resolution (i.e., what is video super-resolution); the description of evaluation metrics (i.e., how (video) superresolution performs); the introduction of optical flow based video super-resolution; the investigation of using optical flow to capture temporal dependency for video super-resolution. Prominently, we give an in-depth study of the deep learning based video super-resolution method, where some representative algorithms are analyzed

show abstract

Section: Discussionmentioning

confidence: 99%

Optical Flow for Video Super-Resolution: A Survey

Tu¹,

Li²,

Xie³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In [31], the famous two-stream architecture is devised by applying two 2D CNN architectures separately to RGB frames and optical flows for action recognition. The idea of two-steam architecture is also explored from the perspective of knowledge distillation [5,33,45]. Moreover, Wang et al [40] devise a self-supervised pretext task by estimating the motion in unlabeled videos.…”

Section: Related Workmentioning

confidence: 99%

Motion-Focused Contrastive Learning of Video Representations

Li¹,

Zhang²,

Qiu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Motion, as the most distinct phenomenon in a video to involve the changes over time, has been unique and critical to the development of video representation learning. In this paper, we ask the question: how important is the motion particularly for self-supervised video representation learning. To this end, we compose a duet of exploiting the motion for data augmentation and feature learning in the regime of contrastive learning. Specifically, we present a Motion-focused Contrastive Learning (MCL) method that regards such duet as the foundation. On one hand, MCL capitalizes on optical flow of each frame in a video to temporally and spatially sample the tubelets (i.e., sequences of associated frame patches across time) as data augmentations. On the other hand, MCL further aligns gradient maps of the convolutional layers to optical flow maps from spatial, temporal and spatio-temporal perspectives, in order to ground motion information in feature learning. Extensive experiments conducted on R(2+1)D backbone demonstrate the effectiveness of our MCL. On UCF101, the linear classifier trained on the representations learnt by MCL achieves 81.91% top-1 accuracy, outperforming ImageNet supervised pre-training by 6.78%. On Kinetics-400, MCL achieves 66.62% top-1 accuracy under the linear protocol. Code is available at https://github.com/YihengZhang-CV/MCL-Motion-Focused-Contrastive-Learning.

show abstract

“…Lee et al [37] proposed to leverage the privileged information from ground-truth images and distill the knowledge by minimizing the distance between the features of the teacher network and the student network. Xiao et al [38] proposed an effective knowledge-distillation method for video super-resolution, which enforces the spatial and temporal characteristics of the teacher network and student network to be consistent. All these distillation-based super-resolution models require the network topology of the teacher network and the student network to be consistent.…”

Section: Knowledge Transfer For Image and Video Super-resolutionmentioning

confidence: 99%

“…We further compare our method with two deep superresolution models, i.e, FAKD [36] and FDVDNet [38], which are based on knowledge distillation. The compared methods distill prior knowledge from the teacher network based on the feature map generated from the intermediate layers.…”

Section: E Ablation Study On Knowledge Transfermentioning

confidence: 99%

Online Video Super-Resolution with Convolutional Kernel Bypass Graft

Xiao¹,

Jiang²,

Zheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning-based models have achieved remarkable performance in video super-resolution (VSR) in recent years, but most of these models are less applicable to online video applications. These methods solely consider the distortion quality and ignore crucial requirements for online applications, e.g., low latency and low model complexity. In this paper, we focus on online video transmission, in which VSR algorithms are required to generate high-resolution video sequences frame by frame in real time. To address such challenges, we propose an extremely low-latency VSR algorithm based on a novel kernel knowledge transfer method, named convolutional kernel bypass graft (CKBG). First, we design a lightweight network structure that does not require future frames as inputs and saves extra time costs for caching these frames. Then, our proposed CKBG method enhances this lightweight base model by bypassing the original network with "kernel grafts", which are extra convolutional kernels containing the prior knowledge of external pretrained image SR models. In the testing phase, we further accelerate the grafted multi-branch network by converting it into a simple single-path structure. Experiment results show that our proposed method can process online video sequences up to 110 FPS, with very low model complexity and competitive SR performance.

show abstract

Space-Time Distillation for Video Super-Resolution

Cited by 38 publications

References 37 publications

Optical Flow for Video Super-Resolution: A Survey

Optical Flow for Video Super-Resolution: A Survey

Motion-Focused Contrastive Learning of Video Representations

Online Video Super-Resolution with Convolutional Kernel Bypass Graft

Contact Info

Product

Resources

About