Uformer: A General U-Shaped Transformer for Image Restoration

Abstract

In this paper, we present Uformer, an effective and efficient Transformer-based architecture, in which we build a hierarchical encoder-decoder network using the Transformer block for image restoration. Uformer has two core designs to make it suitable for this task. The first key element is a local-enhanced window Transformer block, where we use non-overlapping window-based self-attention to reduce the computational requirement and employ the depth-wise convolution in the feedforward network to further improve its potential for capturing local context. The second key element is that we explore three skip-connection schemes to effectively deliver information from the encoder to the decoder. Powered by these two designs, Uformer enjoys a high capability for capturing useful dependencies for image restoration. Extensive experiments on several image restoration tasks demonstrate the superiority of Uformer, including image denoising, deraining, deblurring and demoireing. We expect that our work will encourage further research to explore Transformer-based architectures for low-level vision tasks. The code and models will be available at https://github.com/ZhendongWang6/Uformer.

VRT: A Video Restoration Transformer

“…Recently, Transformer-based models [38,67,83,90] have achieved promising performance in various vision tasks, such as image recognition [6,14,21,39,[50][51][52]52,75,90] and image restoration [11,40,89]. Some methods have tried to use Transformer for video modelling by extending the attention mechanism to the temporal dimension [2,3,38,53,60].…”

MAXIM: Multi-Axis MLP for Image Processing

“…Such a strategy will inevitably cause patch boundary artifacts when applied on larger images using crop-ping [14]. Local-attention based Transformers [51,95] ameliorate this issue, but they are also constrained to have limited sizes of receptive field, or to lose non-locality [23,91], which is a compelling property of Transformers and MLP models relative to hierarchical CNNs.…”

“…Advanced components developed for high-level vision tasks have been brought into lowlevel vision tasks as well. Residual and dense connections [42,93,93,117,118], the multi-scale feature learning [19,40,95], attention mechanisms [64,89,107,108,118],…”

“…The authors of [9] presented a spatial-temporal convolutional self-attention network that exploits local information for video super-resolution. More recently, Swin-IR [51] and UFormer [95] apply efficient window-based local attention models on a range of image restoration tasks. MLP vision models.…”

“…We present, to the best of our knowledge, the first effective general-purpose MLP architecture for low-level vision, which we call Multi-AXIs MLP for image processing (MAXIM). Unlike previous low-level Transformers [9,14,51,95], MAXIM has several desired properties, making it intriguing for image processing tasks. First, MAXIM expresses global receptive fields on arbitrarily large images with linear complexity; Second, it directly supports arbitrary input resolutions, i.e., being fully-convolutional; Lastly, it provides a balanced design of local (Conv) and global (MLP) blocks, outperforming SOTA methods without the necessity for large-scale pre-training [14].…”