Frame and Feature-Context Video Super-Resolution

Abstract

For video super-resolution, current state-of-the-art approaches either process multiple low-resolution (LR) frames to produce each output high-resolution (HR) frame separately in a sliding window fashion or recurrently exploit the previously estimated HR frames to super-resolve the following frame. The main weaknesses of these approaches are: 1) separately generating each output frame may obtain high-quality HR estimates while resulting in unsatisfactory flickering artifacts, and 2) combining previously generated HR frames can produce temporally consistent results in the case of short information flow, but it will cause significant jitter and jagged artifacts because the previous super-resolving errors are constantly accumulated to the subsequent frames. In this paper, we propose a fully end-to-end trainable frame and feature-context video super-resolution (FFCVSR) network that consists of two key sub-networks: local network and context network, where the first one explicitly utilizes a sequence of consecutive LR frames to generate local feature and local SR frame, and the other combines the outputs of local network and the previously estimated HR frames and features to super-resolve the subsequent frame. Our approach takes full advantage of the inter-frame information from multiple LR frames and the context information from previously predicted HR frames, producing temporally consistent highquality results while maintaining real-time speed by directly reusing previous features and frames. Extensive evaluations and comparisons demonstrate that our approach produces state-of-the-art results on a standard benchmark dataset, with advantages in terms of accuracy, efficiency, and visual quality over the existing approaches.

Video Super-Resolution Based on Generative Adversarial Network and Edge Enhancement

“…Sajjadi et al [43] proposed frame-recurrent video super-resolution (FRVSR), which repeatedly using previously estimated SR frames to recover subsequent frames. In addition to reusing the reconstructed HR frames, frame and feature-context video super-resolution (FFCVSR) [44] was proposed to exploit the features of the previous frame repeatedly. Likewise, Wang et al [45] proposed learning for video super-resolution through HR optical flow estimation (SOF-VSR), which innovatively reconstructed highresolution optical flow instead of estimating the optical flow among low-resolution frames to improve the accuracy of motion compensation.…”

Space-Angle Super-Resolution for Multi-View Images

“…They utilize optical flow [22,23,47] or deformable convolution [48,49] for explicit or implicit temporal alignment. Besides, [56] and [17] proposes recurrent neural networks to obtain long-distant HF details.…”

Local-Global Fusion Network for Video Super-Resolution

“…Sajjadi et al [20] introduced the frame-recurrent video superresolution (FRVSR) approach to restore an HR frame from the current and the former LR frames, where the former one is warped using current predicted optical flow. Yan et al [21] proposed the frame and feature-context video superresolution (FFCVSR) approach, where the former information is replaced with the current ones after several iterations to reduce the amassing deviation. Zhu et al [22] put forward the residual invertible spatio-temporal network (RISTN) for video super-resolution, which consists of a residual dense connected LSTM to capture the dependencies among frames.…”

“…However, stacking numerous information can cause great pressure on the following components, making the model unable to fully utilize effective features. In order to overcome such drawbacks, several methods [21]- [23] attempted to integrate the current features with one neighbor in front and one behind, while some others [29]- [31] devoted to combine arbitrary neighboring frames with the reference frame directly. The former imposes a strong coherence between adjacent features and the latter involves abundant temporal information to complement the central features.…”