UPSNet: Unsupervised Pan-Sharpening Network With Registration Learning Between Panchromatic and Multi-Spectral Images

Abstract

Recent advances in deep learning have shown impressive performances for pan-sharpening. Pan-sharpening is the task of enhancing the spatial resolution of a multi-spectral (MS) image by exploiting the high-frequency information of its corresponding panchromatic (PAN) image. Many deep-learning-based pan-sharpening methods have been developed recently, surpassing the performances of traditional pansharpening approaches. However, most of them are trained in lower scales using misaligned PAN-MS training pairs, which has led to undesired artifacts and unsatisfying visual quality. In this paper, we propose an unsupervised learning framework with registration learning for pan-sharpening, called UPSNet. UPSNet can be effectively trained in the original scales, and implicitly learns the registration between PAN and MS images without any dedicatedly designed registration module involved. Additionally, we design two novel loss functions for training UPSNet: a guided-filter-based color loss between network outputs and aligned MS targets; and a dual-gradient detail loss between network outputs and PAN inputs. Extensive experimental results show that our UPSNet can generate pan-sharpened images with remarkable improvements in terms of visual quality and registration, compared to the state-of-the-art methods.

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

“…For example, Ma et al (2020) utilize a discriminator to preserve the spatial information using a gradient regularization between the input Pan and spectrally degraded version of the output from generator. The effective loss functions include gradient loss (Seo et al, 2020), perceptual loss (Zhou et al, 2020), and non-reference loss (Zhou et al, 2021a;Luo et al, 2020).…”

Direct Alignment Of Narrow Field-Of-View Hyperspectral Data And Full-View Rgb Image

“…These works consider geometric alignment models from in-plane translation only [18,19,20] to rotation and projective homography [9]. But while successfully generating high resolution hyperspectral images, they are only robust to very slight misalignment of the input image pairs of similar viewpoint and FoV, i.e.…”

DPAFNet: A Multistage Dense-Parallel Attention Fusion Network for Pansharpening

“…Ref. [16] employed registration learning in pansharpening (UPSNet) to avoid dedicated registering the source images. Ref.…”

“…On the other hand, the source images will suffer from spatial destruction due to the decline of the resolution. Such that various unsupervised learning-based (UL) methods are developed to achieve the pansharpening task, such as [15][16][17][18][19], etc. Specifically, these methods can be designed as an encoder-decoder, where the former is to extract multi-level features, whereas the latter is employed to reconstruct the HRMS.…”