Fast Spatio-Temporal Residual Network for Video Super-Resolution

Li, Sheng; He, Fenghua; Du, Bo; Zhang, Lefei; Xu, Yonghao; Tao, Dacheng

doi:10.1109/cvpr.2019.01077

Cited by 134 publications

(73 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These methods generally have good time efficiency, but because they do not explicitly align images, their alignment effect is difficult to quantitatively analyze. Long short-term memory (LSTM) networks [66] and 3D convolutions [67], [68] are also widely used in the field of video reconstruction. Considering that they do not register images or features, we do not compare these algorithms in this paper.…”

Section: ) Image Superresolutionmentioning

confidence: 99%

Split-Attention Multiframe Alignment Network for Image Restoration

Liu

Feng

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Image registration (or image alignment), the problem of aligning multiple images with relative displacement, is a crucial step in many multiframe image restoration algorithms. To solve the problem that most existing image registration approaches can only align two images in one inference, we propose a splitattention multiframe alignment network (SAMANet). Pixel-level displacements between multiple images are first estimated at low-resolution scales and then refined gradually with the increase in feature resolution. To better integrate the interframe information, we present a split-attention module (SAM) and a dot-product attention module (DPAM), which can adaptively rescale the cost volume features and optical flow features according to the similarity between features from different images. The experimental results demonstrate the superiority of our SAMANet over state-of-the-art image registration methods in terms of both accuracy and robustness. To solve the ''ghosting effect'' caused by pixelwise registration, we designed two ''ghost'' removal modules: warping repetition detection module (WRDM) and attention fusion module (AFM). WRDM detects ''ghost'' regions during the image warping process without increasing the time complexity of the registration algorithm. AFM uses an attention mechanism to rescale the aligned images and enables the registration network and the subsequent image restoration networks to be trained jointly. To validate the strengths of the proposed approaches, we apply SAMANet, WRDM and AFM to three image/video restoration tasks. Extensive evaluations demonstrate that the proposed methods can enhance the performance of image restoration algorithms and outperform the other compared registration algorithms.

show abstract

Section: ) Image Superresolutionmentioning

confidence: 99%

Split-Attention Multiframe Alignment Network for Image Restoration

Liu

Feng

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The pipeline architecture follows feature extraction, feature alignment, fusion and frame reconstruction. Recently, Wang et al [23] reported that the feature alignment and fusion are the crucial steps when the video frames have more occlusion, motion and blurriness. Hence, in order to generate the high-quality HR output, we also focus on the feature alignment and fusion.…”

Section: Align-filter and Learn Video Super Resolution Using Deep Learnmentioning

confidence: 99%

“…3D-Convolution is known as a promising technique for this purpose but due to increased computational complexity the performance of 3D convolutions may degrade. In order to solve this issue, Li et al [23] presented spatio-temporal residual network. In this process, spatio-temporal residual block (STRB) is proposed which divides the 3D filters to reduce the dimension.…”

Section: Literature Surveymentioning

confidence: 99%

Align-Filter & Learn Video Super Resolution using Deep learning (AFLVSR)

2019

IJRTE

View full text Add to dashboard Cite

In this article, we focused on the super-resolution technique in computer vision applications. During last decade, image super-resolution techniques have been introduced and adopted widely in various applications. However, increasing demand of high quality multimedia data has lead towards the high resolution data streaming. The conventional techniques which are based on the image super-resolution are not suitable for multi-frame SR. Moreover, the motion estimation, motion compensation, spatial and temporal information extraction are the well-known challenging issues in video super-resolution field. In this work, we address these issues and developed deep learning based novel architecture which performs feature alignment, filtering the image using deep learning and estimates the residual of low-resolution frames to generate the high-resolution frame. The proposed approach is named as Align-Filter & Learn Video Super resolution using Deep learning (AFLVSR). We have conducted and extensive experimental analysis which shows a significant improvement in the performance when compared with the state-of-art video SR techniques.

show abstract

“…This method enhances performance while maintains a low computational load. 15 Another similar study has suggested spatio-temporal subpixel convolution networks. These networks effectively exploit temporal redundancies and therefore improve reconstruction accuracy with preserving real-time speed.…”

Section: Introductionmentioning

confidence: 96%

“…12 In recent years, the methods based on sparse representation have been used for the echocardiography temporal SR based on the post-processing techniques. 13,14 In recent times, some studies have used a combination of temporal and spatial information in a variety of ways for video SR. [15][16][17][18][19] Deep learning-based video SR methods are the topics of interest in this field, which have promising performances. For example, a novel fast spatio-temporal residual network (FSTRN) is offered to adopt 3D convolutions for the temporal SR task on natural images.…”

Section: Introductionmentioning

confidence: 99%

Temporal Super-resolution of Ultrasound Imaging Using Matrix Completion

2020

View full text Add to dashboard Cite

The temporal super-resolution of the dynamic ultrasound imaging, a means to observe rapid heart movements, is considered an important subject in medical diagnosis of cardiac conditions. Here, a new technique based on the acquisition scheme using the matrix completion (MC) theory is offered for the temporal super-resolution of the two-dimensional (2D) and three-dimensional (3D) ultrasound imaging. MC mentions the problem of completing a low-rank matrix when only a subset of its elements can be observed. Here, the lower scan lines are acquired. Whereby, the proposed method uses temporal and spatial information of the radio frequency (RF) image sequences for the reconstruction of skipped RF lines. This is performed using the construction of the MC images and then reconstruction of them by the MC theory. The results of the proposed method are compared with the compressive sensing (CS) reconstruction methods. The qualitative and quantitative evaluations of 2D and 3D data demonstrate that in the proposed method, which uses the spatial and temporal relation of RF images and the MC theory, the reconstruction is more accurate, and the reconstruction error is lower. The computational complexity of this method is very low. It also does not require hardware adjustments. Therefore, it can be easily implemented in current ultrasound-imaging devices with the frame-rate enhancement. For instance, the frame rate up to two times the original sequence is feasible using the proposed methods, while root mean square error is decreased by about 35% and 30% for 2D and 3D data, respectively, compared with the CS reconstruction method.

show abstract

Fast Spatio-Temporal Residual Network for Video Super-Resolution

Cited by 134 publications

References 37 publications

Split-Attention Multiframe Alignment Network for Image Restoration

Split-Attention Multiframe Alignment Network for Image Restoration

Align-Filter & Learn Video Super Resolution using Deep learning (AFLVSR)

Temporal Super-resolution of Ultrasound Imaging Using Matrix Completion

Contact Info

Product

Resources

About