Rethinking Alignment in Video Super-Resolution Transformers

Abstract

The alignment of adjacent frames is considered an essential operation in video super-resolution (VSR). Advanced VSR models, including the latest VSR Transformers, are generally equipped with well-designed alignment modules. However, the progress of the self-attention mechanism may violate this common sense. In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations. Our experiments show that: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. These observations indicate that we can further improve the performance of VSR Transformers simply by removing the alignment module and adopting a larger attention window. Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions. Therefore, we propose a new and efficient alignment method called patch alignment, which aligns image patches instead of pixels. VSR Transformers equipped with patch alignment could demonstrate state-of-the-art performance on multiple benchmarks. Our work provides valuable insights on how multi-frame information is used in VSR and how to select alignment methods for different networks/datasets. Codes and models will be released at https://github.com/XPixelGroup/RethinkVSRAlignment.Recently, the general paradigm of vision network design has gradually shifted from CNNs to Transformers [9,22,7]. Unlike the locality property of CNNs, the self-attention operation in Transformers is very efficient for processing elements with spatially long-term distribution. VSR is no exception: Transformers have also been introduced into VSR for better results [3,21,23]. However,

Efficient Image Super-Resolution using Vast-Receptive-Field Attention

“…The first effort we made in this paper was to introduce the large receptive field design into the attention mechanism. This is in line with other recent design trends using large kernel sizes [16], as well as the design principles of transformers [9,36,44]. We show the advantages of using large kernel convolutions in the attention branch.…”

“…Vision Transformers [15] rely on attention mechanisms to achieve excellent performance. Many works have proved that introducing large receptive fields and local windows [9,44] in the attention branch improves the SR effect. However, many advanced design ideas have not been verified in designing the attention mechanism for convolutional lightweight SR networks.…”

A lightweight image super-resolution network based on large receptive field information distillation

“…Second, we rethink the network structure of RFDN, simplify it and introduce large convolution kernels in the branches, following the design principles of recent studies on large-scale convolution kernels [13]. Some works have shown that introducing large receptive field convolution kernels and local windows [14,15] in the module branches can improve the performance of SR networks, which is also verified in this paper. Third, considering that large-scale convolution kernels bring performance gain at the cost of parameter size, we use depthwise separable convolution to split large convolution kernels, and implement large receptive field convolution operations by depthwise separable convolution and depthwise separable dilated convolution.…”

Masked Image Training for Generalizable Deep Image Denoising

“…Vision Transformers treat input pixels as tokens and use self-attention operations to process interactions between these tokens. Inspired by the success of vision Transformers, many attempts have been made to employ Transformers for low-level vision tasks [10,14,15,46,63,68,71,75,78,79] During the development of these models, the noise pattern used for training is often consistent with the testing one. The factor that determines its denoising performance is the fitting ability of the network, in other words, the ability of the network to overfit to the training noise.…”