Super-resolution of Omnidirectional Images Using Adversarial Learning

Abstract

An omnidirectional image (ODI) enables viewers to look in every direction from a fixed point through a headmounted display providing an immersive experience compared to that of a standard image. Designing immersive virtual reality systems with ODIs is challenging as they require high resolution content. In this paper, we study super-resolution for ODIs and propose an improved generative adversarial network based model which is optimized to handle the artifacts obtained in the spherical observational space. Specifically, we propose to use a fast PatchGAN discriminator, as it needs fewer parameters and improves the super-resolution at a fine scale. We also explore the generative models with adversarial learning by introducing a spherical-content specific loss function, called 360-SS. To train and test the performance of our proposed model we prepare a dataset of 4500 ODIs. Our results demonstrate the efficacy of the proposed method and identify new challenges in ODI superresolution for future investigations.

360° Single Image Super Resolution via Distortion-Aware Network and Distorted Perspective Images

“…Single-image super-resolution (SISR) aims to enhance the resolution of an image (e.g., 4×). In this work, we propose a novel 360°SISR algorithm that recovers more accurate geometric structures (e.g., characters on the floor) than the state-of-the-arts perspective [4] and 360° [6] SISR algorithms.…”

“…the SR network for perspective images can be applied to the 360°domain even with little or no treatment for the projection distortion, a recent study [9] showed that the GAN-based SISR techniques used in [6] tend to degrade the corresponding PSNR values. Therefore, evaluation based on the reconstruction error in [6] has not revealed the effect of the distortionaware treatment for perspective to 360°domain adaption. The goal of this work is to explore the better transfer from the perspective SISR to the 360°SISR.…”

“…We used 44 test images from Panoramic Object Detection Dataset (PODD) [16] whose image size is 2000 × 4000. We compared our method against 360SR [6] as the current stateof-the-art 360°SISR algorithm and naïve RCAN [4] implementation as one of the current state-of-the-art perspective SISR algorithms. We used four objective metrics: PSNR, SSIM [20], WS-PSNR [15], and WS-SSIM [21] to quantify the reconstruction accuracy of Y channel in YCbCr space.…”

“…For the perspective image, the single-image super-resolution (SISR) approach has been more popular than its multi-image counterpart because of its efficiency, and more importantly, its compatibility with recently developed deep-learning-based techniques [2,3,4]. However, less attention has been paid to the SISR for 360°images, and to the best of our knowledge, only two works [5,6] tackled this problem. Fakour-Sevom et al [5] directly applied the existing perspective SISR network [7], while Ozcinar et al [6] applied conditional adversarial neural networks [2] trained on the SUN 360 Panorama Database [8] with the distortion-aware 360-SS loss to deal with the distortion caused by the sphere-to-plane projection.…”

“…However, less attention has been paid to the SISR for 360°images, and to the best of our knowledge, only two works [5,6] tackled this problem. Fakour-Sevom et al [5] directly applied the existing perspective SISR network [7], while Ozcinar et al [6] applied conditional adversarial neural networks [2] trained on the SUN 360 Panorama Database [8] with the distortion-aware 360-SS loss to deal with the distortion caused by the sphere-to-plane projection. Though these two works [5,6] have revealed that Original 360°image (4×)…”