Video Super-Resolution with Recurrent Structure-Detail Network

Isobe, Takashi; Jia, Xu; Gu, Shuhang; Li, Songjiang; Wang, Shengjin; Tian, Qi

doi:10.1007/978-3-030-58610-2_38

Cited by 175 publications

(162 citation statements)

References 27 publications

Supporting

Mentioning

162

Contrasting

Order By: Relevance

“…Multiple-image SR is closely related with video SR [13], which can be handled using recurrent networks [14]. However, such techniques are based on explicit or implicit assumptions concerned with the input stream, hence they are not applicable in all MISR scenarios.…”

Section: Related Workmentioning

confidence: 99%

A Graph Neural Network For Multiple-Image Super-Resolution

Tarasiewicz

Nalepa

Kawulok

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Super-resolution consists in reconstructing a high-resolution image from single or multiple low-resolution observations. Deep learning has been reported extremely successful for single-image superresolution, but its applications to the multiple-image scenarios are limited due to the challenges that arise from feeding a network with a stack of images with sub-pixel translations. In this paper, we introduce Magnet-a new graph neural network that benefits from representing the input low-resolution images as a graph. This enables us to exploit the sub-pixel shifts among the input images while preserving the original low-resolution pixel values for feature extraction and information fusion. Despite a relatively simple architecture, Magnet outperforms the state-of-the-art methods for multiple-image superresolution, and due to the flexible graph representation, it allows for using a variable number of low-resolution images for reconstruction.

show abstract

Section: Related Workmentioning

confidence: 99%

A Graph Neural Network For Multiple-Image Super-Resolution

Tarasiewicz

Nalepa

Kawulok

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…However, these CNN works are restricted by the concept of the 2D convolution in the spatial domain. A few pieces of works study the joint temporal and spatial feature extraction specifically for video SR [37], [40]- [44], [57], [58], i.e., group convolution and recurrent convolution. One of the straightforward ways is 3D convolution.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, various methods have been proposed to resolve the video SR problem. Based on whether using the temporal information or not, we can classify the video SR methods into two categories: single image SR [1]- [26] and multi-frame SR [27]- [44], [51]- [58].…”

Section: Introductionmentioning

confidence: 99%

Efficient Video Super-Resolution via Hierarchical Temporal Residual Networks

2021

View full text Add to dashboard Cite

Super-Resolving (SR) video is more challenging compared with image super-resolution because of the demanding computation time. To enlarge a low-resolution video, the temporal relationship among frames must be fully exploited. We can model video SR as a multi-frame SR problem and use deep learning methods to estimate the spatial and temporal information. This paper proposes a lighter residual network, based on a multi-stage back projection for multi-frame SR. We improve the back projection based residual block by adding weights for adaptive feature tuning, and add global & local connections to explore deeper feature representation. We jointly learn spatial-temporal feature maps by using the proposed Spatial Convolution Packing scheme as an attention mechanism to extract more information from both spatial and temporal domains. Different from others, our proposed network can input multiple low-resolution frames to obtain multiple super-resolved frames simultaneously. We can then further improve the video SR quality by self-ensemble enhancement to meet videos with different motions and distortions. Results of much experimental work show that our proposed approaches give large improvement over other state-of-the-art video SR methods. Compared to recent CNN based video SR works, our approaches can save, up to 60% computation time and achieve 0.6 dB PSNR improvement.

show abstract

“…Compared with single image restoration, the key challenge of video restoration lies in how to make full use of neighboring highly-related but misaligned supporting frames for the reconstruction of the reference frame. Illustrative comparison of sliding window-based models (1a, e.g., [37,81,88]), recurrent models (1b, e.g., [7,9,18,23,25]) and the proposed parallel VRT model (1c). Green and blue circles denote low-quality (LQ) input frames and high-quality (HQ) output frames, respectively.…”

Section: Introductionmentioning

confidence: 99%

“…Existing video restoration methods can be mainly divided into two categories: sliding window-based methods [4,24,26,34,37,71,81,88,110] and recurrent methods [7,9,18,22,23,25,27,45,59,66,70,109]. As shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

VRT: A Video Restoration Transformer

Liang¹,

Cao²,

Yi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video restoration (e.g., video super-resolution) aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which either is restricted by frame-by-frame restoration or lacks longrange modelling ability. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction and long-range temporal dependency modelling abilities. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal mutual self attention (TMSA) and parallel warping. TMSA divides the video into small clips, on which mutual attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on three tasks, including video super-resolution, video deblurring and video denoising, demonstrate that VRT outperforms the state-of-the-art methods by large margins (up to 2.16dB) on nine benchmark datasets.

show abstract

Video Super-Resolution with Recurrent Structure-Detail Network

Cited by 175 publications

References 27 publications

A Graph Neural Network For Multiple-Image Super-Resolution

A Graph Neural Network For Multiple-Image Super-Resolution

Efficient Video Super-Resolution via Hierarchical Temporal Residual Networks

VRT: A Video Restoration Transformer

Contact Info

Product

Resources

About