Pre-Trained Image Processing Transformer

Chen, Hanting; Wang, Yunhe; Guo, Tianyu; Xu, Chang; Deng, Yiping; Liu, Zhenhua; Ma, Siwei; Xu, Chunjing; Xu, Chao; Gao, Wen

doi:10.1109/cvpr46437.2021.01212

Cited by 1,167 publications

(541 citation statements)

References 50 publications

Supporting

Mentioning

430

Contrasting

Unclassified

Order By: Relevance

“…For image restoration, IPT [44] jointly trains standard Transformer blocks with multi-tails and multiheads on multiple low-level vision tasks. However, their framework still needs pre-training on a large-scale synthesized dataset and requires multi-task learning for good performance.…”

Section: Related Workmentioning

confidence: 99%

Uformer: A General U-Shaped Transformer for Image Restoration

Wang¹,

Cun²,

Bao³

et al. 2021

Preprint

104

View full text Add to dashboard Cite

In this paper, we present Uformer, an effective and efficient Transformer-based architecture, in which we build a hierarchical encoder-decoder network using the Transformer block for image restoration. Uformer has two core designs to make it suitable for this task. The first key element is a local-enhanced window Transformer block, where we use non-overlapping window-based self-attention to reduce the computational requirement and employ the depth-wise convolution in the feedforward network to further improve its potential for capturing local context. The second key element is that we explore three skip-connection schemes to effectively deliver information from the encoder to the decoder. Powered by these two designs, Uformer enjoys a high capability for capturing useful dependencies for image restoration. Extensive experiments on several image restoration tasks demonstrate the superiority of Uformer, including image denoising, deraining, deblurring and demoireing. We expect that our work will encourage further research to explore Transformer-based architectures for low-level vision tasks. The code and models will be available at https://github.com/ZhendongWang6/Uformer.

show abstract

Section: Related Workmentioning

confidence: 99%

Uformer: A General U-Shaped Transformer for Image Restoration

Wang¹,

Cun²,

Bao³

et al. 2021

Preprint

104

View full text Add to dashboard Cite

show abstract

“…Recently, several attempts use Transformer to solve SR. For example, TTSR [35] proposes a texture Transformer by transferring HR textures from the reference image to the LR image. IPT [5] develops a new pre-trained model to study the low-level computer vision task, including SR. However, it is non-trivial and difficult to directly extend these Transformer-based image SR methods to VSR.…”

Section: Related Workmentioning

confidence: 99%

“…First, while the locality is well-known to be crucial for VSR, the fully connected self-attention (FCSA) layer neglects to leverage such information in a video sequence. Typically, most existing vision Transformer methods (e.g., ViT [8] and IPT [5]) split an image into several patches or tokens, which may damage the local spatial information [17] to some extent since the contents (e.g., lines, edges, shapes, and even objects) are divided into different tokens. In addition, this layer focuses on global interaction between the token embeddings by using several fully connected layers to compute attention maps which are irrelevant to local information.…”

Section: Introductionmentioning

confidence: 99%

Video Super-Resolution Transformer

Cao¹,

Li²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Video super-resolution (VSR), with the aim to restore a high-resolution video from its corresponding low-resolution version, is a spatial-temporal sequence prediction problem. Recently, Transformer has been gaining popularity due to its parallel computing ability for sequence-to-sequence modeling. Thus, it seems to be straightforward to apply the vision Transformer to solve VSR. However, the typical block design of Transformer with a fully connected self-attention layer and a tokenwise feed-forward layer does not fit well for VSR due to the following two reasons. First, the fully connected self-attention layer neglects to exploit the data locality because this layer relies on linear layers to compute attention maps. Second, the token-wise feed-forward layer lacks the feature alignment which is important for VSR since this layer independently processes each of the input token embeddings without any interaction among them. In this paper, we make the first attempt to adapt Transformer for VSR. Specifically, to tackle the first issue, we present a spatial-temporal convolutional self-attention layer with a theoretical understanding to exploit the locality information. For the second issue, we design a bidirectional optical flow-based feed-forward layer to discover the correlations across different video frames and also align features. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our proposed method. The code will be available at https://github.com/caojiezhang/VSR-Transformer.

show abstract

“…: We gain long-range dependencies of image sequences by using transformer structure to learn sequential representation for input images. Different from most vision tasks [38], [40], [47], the inputs of the style transfer task belong to two very different domains, usually including artistic paintings and natural images. Therefore, StyT r 2 has two transformer encoders to encode domain-specific features, which are used to translate a sequence from one domain to another domain in the next stage.…”

Section: B Style Transfer Transformermentioning

confidence: 99%

“…Therefore, instead of recurrent neural networks [31], transformer is widely used in various NLP tasks [32]- [37]. Inspired by breakthrough of transformer in NLP, many researchers put forward vision transformers for various image tasks, including object detection [38]- [40], segmentation [41], [42], image classification [43]- [46], image processing and genetation [19], [45], [47]. Traditionally CNNs are designed to learn the local corrections within images containing inductive biases.…”

Section: Introductionmentioning

confidence: 99%

StyTr$^2$: Image Style Transfer with Transformers

Deng¹,

Tang²,

Dong³

et al. 2021

Preprint

View full text Add to dashboard Cite

The goal of image style transfer is to render an image with artistic features guided by a style reference while maintaining the original content. Due to the locality and spatial invariance in CNNs, it is difficult to extract and maintain the global information of input images. Therefore, traditional neural style transfer methods are usually biased and content leak can be observed by running several times of the style transfer process with the same reference style image. To address this critical issue, we take long-range dependencies of input images into account for unbiased style transfer by proposing a transformer-based approach, namely StyTr 2 . In contrast with visual transformers for other vision tasks, our StyTr 2 contains two different transformer encoders to generate domain-specific sequences for content and style, respectively. Following the encoders, a multi-layer transformer decoder is adopted to stylize the content sequence according to the style sequence. In addition, we analyze the deficiency of existing positional encoding methods and propose the content-aware positional encoding (CAPE) which is scale-invariant and more suitable for image style transfer task. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed StyTr 2 compared to state-of-the-art CNN-based and flow-based approaches.

show abstract

Pre-Trained Image Processing Transformer

Cited by 1,167 publications

References 50 publications

Uformer: A General U-Shaped Transformer for Image Restoration

Uformer: A General U-Shaped Transformer for Image Restoration

Video Super-Resolution Transformer

StyTr$^2$: Image Style Transfer with Transformers

Contact Info

Product

Resources

About