Multi-Branch Networks for Video Super-Resolution With Dynamic Reconstruction Strategy

Zhang, Dongyang; Shao, Jie; Liang, Zhenwen; Liu, Xueliang; Shen, Heng Tao

doi:10.1109/tcsvt.2020.3044451

Cited by 19 publications

(3 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jo et al [30] predicted dynamics upsampling filters via a dynamic filter generation network, which are then used to filter LR frames for enlarging their spatial resolutions. Zhang et al [31] designed a multibranch network to learn multi-scale spatial features from LR frames to boost the VSR performance. Other methods have utilized the convolutional LSTM [32], generative adversarial network [33], and graph neural network [34] to reconstruct HR frames.…”

Section: B Video Super-resolutionmentioning

confidence: 99%

Enhancing Space-time Video Super-resolution via Spatial-temporal Feature Interaction

Yue¹,

Shi²

2022

Preprint

View full text Add to dashboard Cite

The target of space-time video super-resolution (STVSR) is to increase both the frame rate (also referred to as the temporal resolution) and the spatial resolution of a given video. Recent approaches solve STVSR with end-to-end deep neural networks. A popular solution is to first increase the frame rate of the video; then perform feature refinement among different frame features; and last increase the spatial resolutions of these features. The temporal correlation among features of different frames is carefully exploited in this process. The spatial correlation among features of different (spatial) resolutions, despite being also very important, is however not emphasized. In this paper, we propose a spatial-temporal feature interaction network to enhance STVSR by exploiting both spatial and temporal correlations among features of different frames and spatial resolutions. Specifically, the spatial-temporal frame interpolation module is introduced to interpolate low-and high-resolution intermediate frame features simultaneously and interactively. The spatialtemporal local and global refinement modules are respectively deployed afterwards to exploit the spatial-temporal correlation among different features for their refinement. Finally, a novel motion consistency loss is employed to enhance the motion continuity among reconstructed frames. We conduct experiments on three standard benchmarks, Vid4, Vimeo-90K and Adobe240, and the results demonstrate that our method improves the state of the art methods by a considerable margin. Our codes will be available at https://github.com/yuezijie/STINet-Space-time-Video-Super-resolution.

show abstract

Section: B Video Super-resolutionmentioning

confidence: 99%

Enhancing Space-time Video Super-resolution via Spatial-temporal Feature Interaction

Yue¹,

Shi²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…is considerably impacts the daily dissemination of information-the network speed cannot keep up, and the hard disk cannot store it. erefore, there is an urgent need for an e cient means of information compression to help compress information to improve transmission e ciency and reduce the storage footprint [2]. With the development of high-performance processors, high-de nition screens are becoming more and more popular with the emergence of intelligent devices.…”

Section: Introductionmentioning

confidence: 99%

Research on Super-Resolution Relationship Extraction and Reconstruction Methods for Images Based on Multimodal Graph Convolutional Networks

Xiao

2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

This study constructs a multimodal graph convolutional network model, conducts an in-depth study on image super-resolution relationship extraction and reconstruction methods, and constructs a model of image super-resolution relationship extraction and reconstruction methods based on multimodal graph convolutional networks. In this study, we study the domain adaptation model algorithm based on chart convolutional networks, which constructs a global relevance graph based on all samples using pre-extracted features and performs distribution approximation of sample features in two domains using a diagram convolutional neural network with maximum mean difference loss; with this approach, the model effectively preserves the structural information among the samples. In this study, several comparison experiments are designed based on the COCO and VG datasets; the image space information-based and knowledge graph-based target detection and recognition models substantially improve recognition performance over the baseline model. The super-pixel-based target detection and recognition model can also effectively reduce the number of floating-point operations and the complexity of the model. In this study, we propose a multiscale GAN-based image super-resolution reconstruction algorithm. Aiming at the problems of detail loss or blurring in the reconstruction of detail-rich images by SRGAN, it integrates the idea of the Laplace pyramid to complete the task of multiscale reconstruction of images through staged reconstruction. It incorporates the concept of a discriminative network with patch GAN to effectively improve the recovery effect of graph details and improve the reconstruction quality of images. Using Set5, Set14, BSD100, and Urban100 datasets as test sets, experimental analysis is conducted from objective and subjective evaluation metrics to effectively validate the performance of the improved algorithm proposed in this study.

show abstract

“…θ is the parameters of F (•). To obtain different resolution images in real-scene configuration, I hf and I lf are collected by different optical sensors [6], [7], [8], [9], [10] with various resolution settings, which is different from the traditional image super-resolution paradigm [11], [12], [13], [14], [15], [16] that generates I lf using downsampling techniques. Therefore, compared with the traditional image super-resolution task, RealSR suffers a severer pixel displacement due to the difference between the camera settings to obtain I hf and I lf .…”

Section: Introductionmentioning

confidence: 99%

Criteria Comparative Learning for Real-scene Image Super-Resolution

Shi¹,

Li²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Real-scene image super-resolution aims to restore real-world low-resolution images into their high-quality versions. A typical RealSR framework usually includes the optimization of multiple criteria which are designed for different image properties, by making the implicit assumption that the ground-truth images can provide a good trade-off between different criteria. However, this assumption could be easily violated in practice due to the inherent contrastive relationship between different image properties. Contrastive learning (CL) provides a promising recipe to relieve this problem by learning discriminative features using the triplet contrastive losses. Though CL has achieved significant success in many computer vision tasks, it is non-trivial to introduce CL to RealSR due to the difficulty in defining valid positive image pairs in this case. Inspired by the observation that the contrastive relationship could also exist between the criteria, in this work, we propose a novel training paradigm for RealSR, named Criteria Comparative Learning (Cria-CL), by developing contrastive losses defined on criteria instead of image patches. In addition, a spatial projector is proposed to obtain a good view for Cria-CL in RealSR. Our experiments demonstrate that compared with the typical weighted regression strategy, our method achieves a significant improvement under similar parameter settings.

show abstract

Multi-Branch Networks for Video Super-Resolution With Dynamic Reconstruction Strategy

Cited by 19 publications

References 53 publications

Enhancing Space-time Video Super-resolution via Spatial-temporal Feature Interaction

Enhancing Space-time Video Super-resolution via Spatial-temporal Feature Interaction

Research on Super-Resolution Relationship Extraction and Reconstruction Methods for Images Based on Multimodal Graph Convolutional Networks

Criteria Comparative Learning for Real-scene Image Super-Resolution

Contact Info

Product

Resources

About