Pansharpening via Unsupervised Convolutional Neural Networks

Abstract

Pansharpening is normally utilized to take full advantage of all the available spectral and spatial information that are derived from a low-spatial-resolution (LR) multispectral (MS) image and its associated high-spatial-resolution (HR) panchromatic (PAN) image, respectively, producing a fused MS image with high spectral and spatial resolutions. Many methods have been recently developed based on convolutional neural networks (CNNs) for the pansharpening task, but most of them still have some drawbacks: 1) The information cannot efficiently flow in their simple stacked convolutional architectures, thereby hindering the representation ability of the networks; 2) They are commonly trained using supervised learning, which does not only require an extra effort to produce the simulated training data, but can also lead to scale-related problems in the fusion results. In this paper, we propose a novel unsupervised CNN-based pansharpening method to overcome these limitations. Specifically, we design an iterative network architecture, in which a PANguided strategy and a set of skip connections (SC) are adopted to continuously extract and fuse the features from the input, thus enhancing the information reuse and transmission. Besides, we propose a new loss function for unsupervised training in which the relationships between the input MS and PAN images and the fused MS image are used to design the spatial constrains and spectral consistency, respectively. The typical quality index with noreference (QNR) is also added to this function to further adjust the spectral and spatial qualities. The designed loss function allows the network to be learned only on input images, without any handcrafted labels (reference HR MS image). We evaluated the effectiveness of our designed network architecture and the combined loss function, and the experiments testify that our unsupervised strategy can also obtain promising results with minor spectral and spatial distortions compared with other traditional and supervised methods.

UPSNet: Unsupervised Pan-Sharpening Network With Registration Learning Between Panchromatic and Multi-Spectral Images

“…Qu et al incorporated a self-attention mechanism [32] that estimates spatially varying detail extraction and injection functions. Luo et al also proposed an unsupervised pan-sharpening method [25] with an iterative fusion network. Although these unsupervised PS methods resolved the drawbacks of training in lower scales, none of them considered the inherent misalignment between MS and PAN inputs.…”

PGMAN: An Unsupervised Generative Multiadversarial Network for Pansharpening

“…Wald's protocol has been widely used for assessment of pan-sharpening methods, in which the original MS and PAN images are spatially degraded before feeding into models, the reducing factor being the ratio between their spatial resolutions, and the original MS images are considered as reference images for comparison. Same as the previous works [40], [46], we implement it by blurring the full-resolution datasets using a Gaussian filter and then downsampling them with a scaling factor of 4. Under Wald's protocol, the supervised models can be trained on reduced-resolution images using the original MS images as labels.…”

Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model

“…Besides, there are available unsupervised DL-based methods that exploit the spatial and spectral consistency between HR and LR images. An example is the method in [36] which derives a new loss function for unsupervised training a CNN based on the spatial correlation between the estimated HR and PAN images, and the spectral correlation between the estimated HR and the interpolated LR images.…”

“…They have been proposed for the related remote sensing image fusion, such as pansharpening, hyperspectral, and multispectral image fusion. For pansharpening, examples are the methods in [33] and [34] using generative adversarial networks (GANs), and the methods in [35] and [36] using CNNs with the spectral and spatial constraints of PAN and MS images. For hyperspectral and multispectral image fusion, the fusion strategy [37] utilizes an unsupervised adaption learning, which learns a general image prior by using a deep network and adapts it to a specific hyperspectral image.…”