A Unified Two-Stage Spatial and Spectral Network With Few-Shot Learning for Pansharpening

Abstract

Recently, pan-sharpening methods based on deep learning (DL) have achieved state-of-the-art results. However, current existing DL-based pan-sharpening methods need to be trained repetitively for different satellite sensors to obtain satisfactory fusion performance and therefore require a large number of training images for each satellite. To deal with these issues, in this paper we propose a unified two-stage spatial and spectral network (UTSN) for pan-sharpening. A branch of networks is constructed for each different satellite, in which the spatial enhancement network (SEN) is shared to improve the spatial details in the fused images from different satellites. A spectral adjustment network (SAN) is employed to capture the spectral characteristics of the specific satellite. Through SAN, the spectral information in the intermediate image from SEN is refined to produce the final fusion results. Such a framework can integrate the datasets from different satellites together for sufficient training of SEN. The proposed method is able to achieve promising pan-sharpening results also for a new satellite with limited training images by only learning a new SAN on the few-shot datasets due to the simple but efficient structure of SAN. The experimental results show that the proposed method can produce state-of-the-art fusion results in both the standard and few-shot cases. The source code is publicly available at https://github.com/RSMagneto/UTSN.

UPGAN: An Unsupervised Generative Adversarial Network Based on U-Shaped Structure for Pansharpening

“…Liu et al [37] investigated a network with two branches to carry out fusion in the feature domain, which first encodes input images into high-level feature representations and then reconstructs high-resolution images. A unified two-stage spatial and spectral network has been proposed, called UTSN [38], which contains a spatial enhancement network, which was trained and shared on hybrid data sets, and a spectral adjustment network, which is used to capture the spectral characteristics of a specific satellite. However, it should be noted that supervised learning models generate simulated results with limited real-world applicability; furthermore, the process of training fails to make full use of the original high-resolution information, potentially resulting in scale mismatches.…”