Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Abstract

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image superresolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4× upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method. arXiv:1609.04802v5 [cs.CV]

Cascaded SR-GAN for Scale-Adaptive Low Resolution Person Re-identification

“…1. The general idea behind this formulation is that it allows one to train a generator model G θ G with the goal of fooling a differentiable discriminator D θ D that is trained to distinguish super-resolved images from real images [Ledig et al, 2016].…”

“…In general, super-resolution is commonly modeled based on the pixel-wise MSE . In particular, we introduce the adversarial loss for each generator network k, and exploit the VGG loss for perceptual similarity as [Ledig et al, 2016] …”

“…Following [Ledig et al, 2016], we train the generator networks with pixel-wise loss, VGG loss, and adversarial loss, which together form the generator network loss l SR Gen . For super resolution, we also train the discriminator networks with loss l SR Dis from the perspectives of game theory, and boost the generator networks indirectly.…”

“…SR-GAN [Ledig et al, 2016] proposes a very deep ResNet [He et al, 2016] architecture using the concept of GAN for photo-realistic image super-resolution. It sets a new state-of-the-art for image SR with a fixed upscaling factor (4×).…”

“…During the process of super-resolving, SR methods Johnson et al, 2016; supplement and recover the missing appearance information (Figure 1), so that LR gallery images and the HR probe image can be treated equally. The generative adversarial network (GAN) for image SR [Ledig et al, 2016], i.e., SR-GAN, offers an effective solution to image SR, and is also one of the most competitive SR approaches. Nevertheless, (1) a fixed SR module is not suitable for the scale-adaptive LR scenario.…”