Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Abstract

Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video superresolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal subpixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is endto-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to singleframe models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-theart performance in both accuracy and efficiency.

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

“…Super-Resolution Single image super resolution (SISR) methods upscale individual images, which include traditional methods based on bilinear and bicubic filters and recently-introduced learning-based techniques [26], [28]. SISR methods can be easily extended to support video or multi-frame super-resolution [10], [22], [34]. In the case of multi-camera setup, super-resolution can be performed using some source as a reference [6], [14].…”

“…• Step 3: End-to-End Joint Training. Starting with our pretrained models, we jointly train FlowNet and FusionNet by minimizing the same end-to-end 1 loss in (10). In this step, we set learning rate to 10 −5 for FusionNet and 3×10 −6 for FlowNet over 100k iterations.…”

Globally Guided Progressive Fusion Network for 3D Pancreas Segmentation

“…Between every two consecutive groups, max pooling is used to reduce the spatial resolution of the feature map by half, giving rise to feature maps with 4 different scales. Inspired from [1], our network progressively fuses the slices in the input 3D patch by not performing the convolution operation in the 2 outmost slices in every 3D convolution layer because these two slices are of least relevance to the central slice. We choose T to be the number of 3D convolution layers so that there exists only one slice (the central slice) in the final group of feature maps, E k .…”

Frame and Feature-Context Video Super-Resolution

“…Then, we compare our approach with current state-of-the-art methods on the standard Vid4 benchmark dataset (Liu and Sun 2011) in terms of visual quality, objective metric, temporal consistency, and computational cost. Following (Caballero et al 2017), the evaluation metrics of Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are computed on the brightness channel on the 4-video dataset Vid4. Thirdly, we detail the suppression-updating algorithm for depressing iteration error of high-frequency information.…”

“…To evaluate the performance of our approach on real-world data, following (Caballero et al 2017), a visual comparison result is reported in Fig. 8.…”