Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration

Conde, Marcos V.; Choi, Ui-Jin; Burchi, Maxime; Timofte, Radu

doi:10.1007/978-3-031-25063-7_42

Cited by 62 publications

(30 citation statements)

References 112 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Originally, the Transformer [19] was designed for Natural Language Processing (NLP) applications, which has achieved state-of-the-art performance and become the de-facto standard solution for many NLP tasks [20], such as the cutting-edge and high-profile Chat Generative Pre-trained Transformer (Chat-GPT) [37]. Inspired by the great success of the Transformer in NLP, the Vision Transformer (ViT), a variant of the Transformer designed specifically for image processing, has recently gained much popularity in the computer vision community [18,38]. Different from the CNN structure, the ViT converts 2-D images into 1-D sequences first and then applies the selfattention mechanism for feature extraction.…”

Section: Vision Transformermentioning

confidence: 99%

“…Benefitting from this design, the ViT-based algorithms have demonstrated obvious superiority over CNNs and obtained numerous breakthroughs in fundamental vision tasks, such as image classification [20,21,23], semantic segmentation [24,39,40] and super-resolution [18,41,28,42].…”

Section: Vision Transformermentioning

confidence: 99%

“…Among those revolutionary deep-learning-based superresolution algorithms, most of them are constructed by the Convolutional Neural Network (CNN) [6,16,17]. However, as pointed out by [18], the utilization of CNN for super-resolution has two obvious drawbacks. First, as interactions between images and kernels are content-independent, using the same kernel to restore different image regions may not be the optimal solution.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LSwinSR: UAV Imagery Super-Resolution based on Linear Swin Transformer

Li¹,

Zhao²

2023

Preprint

View full text Add to dashboard Cite

Super-resolution, which aims to reconstruct high-resolution images from low-resolution images, has drawn considerable attention and has been intensively studied in computer vision and remote sensing communities. The super-resolution technology is especially beneficial for Unmanned Aerial Vehicles (UAV), as the amount and resolution of images captured by UAV are highly limited by physical constraints such as flight altitude and load capacity. In the wake of the successful application of deep learning methods in the super-resolution task, in recent years, a series of super-resolution algorithms have been developed. In this paper, for the super-resolution of UAV images, a novel network based on the state-of-the-art Swin Transformer is proposed with better efficiency and competitive accuracy. Meanwhile, as one of the essential applications of the UAV is land cover and land use monitoring, simple image quality assessments such as the Peak-Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) are not enough to comprehensively measure the performance of an algorithm. Therefore, we further investigate the effectiveness of super-resolution methods using the accuracy of semantic segmentation. The code will be available at https://github.com/ lironui/GeoSR.

show abstract

Section: Vision Transformermentioning

confidence: 99%

Section: Vision Transformermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LSwinSR: UAV Imagery Super-Resolution based on Linear Swin Transformer

Li¹,

Zhao²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…SwinIR [30] proposes an efficient transformer-based SR model which fully explores the swin transformer structure, and it outperforms pure convolution networks with fewer parameters and FLOPs. Swin2SR [8] further improves the network structure by introducing the SwinV2 attention, and proposes auxiliary loss and high-frequency loss for the compressed images. However, all these methods cannot satisfy real-time performance.…”

Section: Efficient Image Super-resolutionmentioning

confidence: 99%

Baojia Sun, Gongdao Zhang v Shanghai Huili Group Co, Ltd, Shanghai Huili Flooring Products Co, Ltd

Guo¹,

Lin²,

Weng³

2023

Selected Chinese Cases on the CISG

View full text Add to dashboard Cite

In recent years, videos and images in 720p (HD), 1080p (FHD) and 4K (UHD) resolution have become more popular for display devices such as TVs, mobile phones and VR. However, these high resolution images cannot achieve the expected visual effect due to the limitation of the internet bandwidth, and bring a great challenge for super-resolution networks to achieve real-time performance. Following this challenge, we explore multiple efficient network designs, such as pixel-unshuffle, repeat upscaling, and local skip connection removal, and propose a fast and lightweight super-resolution network. Furthermore, by analyzing the applications of the idea of divide-and-conquer in superresolution, we propose assembled convolutions which can adapt convolution kernels according to the input features. Experiments suggest that our method outperforms all the state-of-the-art efficient super-resolution models, and achieves optimal results in terms of runtime and quality.

show abstract

“…18 Much of the early work on super-resolution dealt with scenarios without compression. However recent works 19,20 address the more realistic scenario where a downsampled image or video is compressed for transmission.…”

Section: Related Workmentioning

confidence: 99%

Switchable CNNs for in-loop restoration and super-resolution for AV2

Joshi,

Chen,

Yoo

et al. 2023

Applications of Digital Image Processing XLVI

View full text Add to dashboard Cite

In video codecs, CNN-based models have shown huge promise in two related tasks: in-loop restoration and frame super-resolution. In our previous work, we presented a framework that uses a common CNN architecture with downloadable model parameters for both these tasks with a preliminary performance study, where encoderside selection of scale factor was left as future work. The advantage of a common architecture with switchable parameters is that a single hardware inference engine can be utilized in all cases of same-resolution and super-resolution restoration, thereby limiting implementation costs. In this paper, we fully integrate this framework into the under-development AV2 video codec from the Alliance for Open Media (AOM). We also implement an algorithm for encoder-side selection of the super-resolution scale factor. With this implementation, we are able to achieve combined compression improvement up to −3.5% (AI) and −3.9% (RA) in BDRATE PSNR-Y and up to −7.8% (AI) and −7.9% (RA) in BDRATE VMAF, with inference cost as low as 1500 MACs/pixel.

show abstract

Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration

Cited by 62 publications

References 112 publications

LSwinSR: UAV Imagery Super-Resolution based on Linear Swin Transformer

LSwinSR: UAV Imagery Super-Resolution based on Linear Swin Transformer

Baojia Sun, Gongdao Zhang v Shanghai Huili Group Co, Ltd, Shanghai Huili Flooring Products Co, Ltd

Switchable CNNs for in-loop restoration and super-resolution for AV2

Contact Info

Product

Resources

About