Dual Attention with the Self-Attention Alignment for Efficient Video Super-resolution

Abstract

1) Background/IntroductionBy selectively enhancing the features extracted from convolution networks, the attention mechanism has shown its effectiveness for low-level visual tasks, especially for image super-resolution (SR). However, due to the spatiotemporal continuity of video sequences, simply applying image attention to a video cannotdoes not seem to obtain good SR results. At present, there is still a lack of suitable attention structure to achieve efficient video SR. 2) MethodsIn this work, building uponbased on the correlation exploration for the dual attention, i.e., -position attention and channel attention, we proposed deep dual attention, underpinned by equipped with self-attention alignment (DASAA), for video SR.Specifically, Firstly, we start by constructing a dual attention module (DAM) to strengthen the acquired spatiotemporal features and adopt a self-attention structure with the morphological mask to achieve attention alignment. Then, on top of based on the attention features, we utilize the up-sampling operation to reconstruct the super-resolved video images, and introduce the LSTM (long short time memory) network to guarantee the coherent consistency of the generated video frames both in temporal and spatial domains temporally and spatially. 3) ResultsExperimental results and comparisons on the actual Youku-VESR dataset and the typical benchmark dataset-Vimeo-90k demonstrate that our proposed approach not only achieves the best video SR effect while but also takinges the least amount of computation. Specifically, in the Youku-VESR dataset, our proposed approach achieves a test PSNR of over PSNR/SSIM metrics is 35.290db and a SSIM of 0.939, respectively. In the Vimeo-90k dataset, the PSNR/SSIM indexes of our approach are individually 32.878db and 0.774. Moreover, the FLOPS (float-point operations per second) of our approach is as low as 6.39G. 4) ConclusionsThe proposed DASAA method surpasses all video SR algorithms in the comparison outperforms all the compared algorithms for video SR. It is also revealedproved that there is no linear relationship between positional attention and channel attention. It 2 suggests that our DASAA with LSTM coherent consistency architecture may have great potential for many low-level vision video applications.