Video Super-Resolution via a Spatio-Temporal Alignment Network

Wen, Weilei; Ren, Wenqi; Shi, Yinghuan; Nie, Yunfeng; Zhang, George; Cao, Xiaochun

doi:10.1109/tip.2022.3146625

Cited by 32 publications

(20 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As it lacked preprocessing before the deformable convolution operation, it had limitations in improving the SR accuracy of the generated HR frame. Wen et al [ 51 ] proposed a spatio-temporal alignment network (STAN) which consists of a filter-adaptive alignment network and an HR image reconstruction network. After the iterative spatio-temporal learning scheme of the filter adaptive alignment network extracts the intermediate feature maps from the input LR frames, a final HR frame is generated using the HR image reconstruction network, which consists of twenty residual channel attention blocks and two up-sampling layers.…”

Section: Related Workmentioning

confidence: 99%

Video Super-Resolution Method Using Deformable Convolution-Based Alignment Network

Lee

Cho

Jun

2022

Sensors

View full text Add to dashboard Cite

With the advancement of sensors, image and video processing have developed for use in the visual sensing area. Among them, video super-resolution (VSR) aims to reconstruct high-resolution sequences from low-resolution sequences. To use consecutive contexts within a low-resolution sequence, VSR learns the spatial and temporal characteristics of multiple frames of the low-resolution sequence. As one of the convolutional neural network-based VSR methods, we propose a deformable convolution-based alignment network (DCAN) to generate scaled high-resolution sequences with quadruple the size of the low-resolution sequences. The proposed method consists of a feature extraction block, two different alignment blocks that use deformable convolution, and an up-sampling block. Experimental results show that the proposed DCAN achieved better performances in both the peak signal-to-noise ratio and structural similarity index measure than the compared methods. The proposed DCAN significantly reduces the network complexities, such as the number of network parameters, the total memory, and the inference speed, compared with the latest method.

show abstract

Section: Related Workmentioning

confidence: 99%

Video Super-Resolution Method Using Deformable Convolution-Based Alignment Network

Lee

Cho

Jun

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Inspired by non-local means denoising algorithm [11], Wang et al [42] proposed a non-local building block which captures long-range dependencies within feature maps. This study prepared the ground for many vision applications, such as classification [33,48], object detection [12,17], image segmentation [21,30], image superresolution [40], and video super-resolution [24,43,44]. Regardless, the idea of attention is fairly new on image alignment algorithms.…”

Section: Attention and Attention-based Alignmentmentioning

confidence: 99%

“…Regardless, the idea of attention is fairly new on image alignment algorithms. Only a handful of studies [7,43,47] are present, most of which do not focus on efficiency and deployment on embedded environments.…”

Section: Attention and Attention-based Alignmentmentioning

confidence: 99%

“…Image alignment (image registration) aims to align or match images to a chosen reference image. This task constitutes an important part of many computer vision problems dealing with multiple images either in space and/or time, such as restoration [41,47], segmentation [21], HDR imaging [45], stereo imaging [40], multi and single-image super-resolution and video super-resolution [24,43,44].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices

Bilecen¹,

Fisne²,

Ayazoğlu³

2022

Preprint

View full text Add to dashboard Cite

Image alignment, also known as image registration, is a critical block used in many computer vision problems. One of the key factors in alignment is efficiency, as inefficient aligners can cause significant overhead to the overall problem. In the literature, there are some blocks that appear to do the alignment operation, although most do not focus on efficiency. Therefore, an image alignment block which can both work in time and/or space and can work on edge devices would be beneficial for almost all networks dealing with multiple images. Given its wide usage and importance, we propose an efficient, cross-attention-based, multipurpose image alignment block (XABA) suitable to work within edge devices. Using cross-attention, we exploit the relationships between features extracted from images. To make cross-attention feasible for real-time image alignment problems and handle large motions, we provide a pyramidal block based cross-attention scheme. This also captures local relationships besides reducing memory requirements and number of operations. Efficient XABA models achieve real-time requirements of running above 20 FPS performance on NVIDIA Jetson Xavier with 30W power consumption compared to other powerful computers. Used as a sub-block in a larger network, XABA also improves multiimage super-resolution network performance in comparison to other alignment methods.

show abstract

“…Similar to [11], [19], [20], we employe Vimeo-90k [21] containing 64612 training samples with spatial resolution 448×256 as the training set. The corresponding LR sequecnces with spatial resolution 112×64 are obtained by bicubically sampling with 4 times.…”

Section: Experiments a Implementation Detailsmentioning

confidence: 99%

Grouped Spatio-Temporal Alignment Network for Video Super-Resolution

Zhang

2022

IEEE Signal Process. Lett.

View full text Add to dashboard Cite

Video super-resolution based on CNNs has recently achieved significant progress. The existing popular superresolution methods usually impose the optical flow between neighboring frames for temporal alignment. However, estimation of accurate optical flow is hard and expends greater computation.To address this problem, we propose a novel grouped spatiotemporal alignment network (GSTAN) that effectively incorporates spatio-temporal information in a hierarchical way. The input sequence is divided into several groups corresponding to different frame rates. These groups provide complementary information, which is helpful to restore missing textures for the reference frame. Specifically, each group employs deformable 3D convolution to incorporate spatio-temporal information, which avoids artifacts from explicit motion estimation. In addition, a Gated-Dconv information filter is proposed to control information flow to focus on the fine details. Finally, these groups provide complementary information, which is integrated with the intergroup fusion module. Extensive experiments have demonstrated our method achieves state-of-the-art SR performance on several benchmark datasets.

show abstract

Video Super-Resolution via a Spatio-Temporal Alignment Network

Cited by 32 publications

References 51 publications

Video Super-Resolution Method Using Deformable Convolution-Based Alignment Network

Video Super-Resolution Method Using Deformable Convolution-Based Alignment Network

Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices

Grouped Spatio-Temporal Alignment Network for Video Super-Resolution

Contact Info

Product

Resources

About