Single satellite imagery simultaneous super-resolution and colorization using multi-task deep neural networks

Abstract

Satellite imagery is a kind of typical remote sensing data, which holds preponderance in large area imaging and strong macro integrity. However, for most commercial space usages, such as virtual display of urban traffic flow, virtual interaction of environmental resources, one drawback of satellite imagery is its low spatial resolution, failing to provide the clear image details. Moreover, in recent years, synthesizing the color for grayscale satellite imagery or recovering the original color of camouflage sensitive regions becomes an urgent requirement for large spatial objects virtual reality interaction. In this work, unlike existing works which solve these two problems separately, we focus on achieving image super-resolution (SR) and image colorization synchronously. Based on multi-task learning, we provide a novel deep neural network model to fulfill single satellite imagery SR and colorization simultaneously. By feeding back the color feature representations into the SR network and jointly optimizing such two tasks, our deep model successfully achieves the mutual cooperation between imagery reconstruction and image colorization.

The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery

“…The authors analyze several sources of satellite imagery for this research and quantify their success in terms of PSNR for an 8× enhancement using a GAN. In another example, [22] use deep neural networks for simultaneous 4× super-resolution and colorization of satellite imagery. Several papers [41,25,35,20,28] modify or leverage SRCNN [7] and/or VDSR [15] to successfully super-resolve Jilin-1, SPOT, Pleiades, Sentinel-2, and Landsat imagery.…”

Deep Learning for Automatic Colorization of Legacy Grayscale Aerial Photographs

“…Convolutions allow retrieval of information from an image, using a sliding window defined by a kernel size and weights associated with each of its cells. More than half of the currently proposed colorization techniques use CNNs for 1) classification [34], [36], [38], [40], [41], [47], [49], [50] and 2) regression [34], [36], [39], [44]. However, regression loss functions, mostly based on Euclidean distance, produce blurry and unsaturated outputs as they tend to minimize the prediction error.…”

“…Furthermore, only a very limited number of works have tackled the problem of colorizing spatial products [47], [50], let alone historical ones. Song et al [47] used a pretrained VGG-16 network to extract high-level features from single-polarization SAR satellite images and reconstruct their full-polarization counterparts.…”

“…Song et al [47] used a pretrained VGG-16 network to extract high-level features from single-polarization SAR satellite images and reconstruct their full-polarization counterparts. Liu et al [50] trained a multitask autoencoder to perform colorization and superresolution on VHR optical satellite imagery. Although both approaches present novel techniques for processing spatial products, they focused on current satellite imagery only, and they are not ideally suited for legacy aerial photographs due to the various domain gaps and the inherent characteristics of these data.…”

“…We converted color images to the CIE Lab color space, as recommended by numerous authors [34], [36], [37], [41], [44]- [46], [50], using Scikit-Image. Chroma is denoted by both a and b.…”