Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Abstract

Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.

Cascaded SR-GAN for Scale-Adaptive Low Resolution Person Re-identification

“…The generator function G θ G is parametrized by θ G , and the discriminator function D θ D is parametrized by θ D . Generally, the target of previous supervised SR algorithms is commonly the minimization of the mean squared error (MSE) [Wang and Bovik, 2009] between the recovered HR image and the ground truth. Besides the MSE loss, SR-GAN also defines a perceptual loss using high-level feature maps of the VGG network [Simonyan and Zisserman, 2014], which makes the super-resolved image and HR reference image perceptually similar.…”

“…Furthermore, a VGG loss based on the ReLU activation layers of the pre-trained 19 layer VGG network, described in [Simonyan and Zisserman, 2014], is exploited for perceptual similarity, measuring on higher semantical level which a naive MSE loss is unable to handle. It should be mentioned that the VGG losses are adopted in all the VGG networks, who share common parameters.…”

“…Generally, the output of VGG network [Simonyan and Zisserman, 2014] is the category of input image. Suppose that the VGG network is able to judge each real imageÎ 3 as a human category.…”

“…LR images are randomly downsampled and selected to construct the gallery set. Cumulative Matching Characteristic (CMC) curves [Wang et al, 2007] were used to calculate the average performance, and the value of CMC@k indicates the percentage of the real match ranked in the top k.…”

“…Person re-identification (REID) is the task of visually matching images of the same person, extracted from nonoverlapping camera views in open surveillance spaces [Wang et al, 2016a]. Since biometric cues, such as face and gait, are usually unreliable or even infeasible in the uncontrolled surveillance environment, the appearance of individuals is The resolutions of these images are significantly different.…”