PanFormer: a Transformer Based Model for Pan-sharpening

Abstract

Pan-sharpening aims at producing a high-resolution (HR) multi-spectral (MS) image from a low-resolution (LR) multispectral (MS) image and its corresponding panchromatic (PAN) image acquired by a same satellite. Inspired by a new fashion in recent deep learning community, we propose a novel Transformer based model for pan-sharpening. We explore the potential of Transformer in image feature extraction and fusion. Following the successful development of vision transformers, we design a two-stream network with the selfattention to extract the modality-specific features from the PAN and MS modalities and apply a cross-attention module to merge the spectral and spatial features. The pan-sharpened image is produced from the enhanced fused features. Extensive experiments on GaoFen-2 and WorldView-3 images demonstrate that our Transformer based model achieves impressive results and outperforms many existing CNN based methods, which shows the great potential of introducing Transformer to the pan-sharpening task. Codes are available at https://github.com/zhysora/PanFormer.

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

“…The first GAN-based pansharpening method designs a two-branch generator network (Liu et al, 2020a), and then different loss functions and new network structures are explored to extract more discriminative features (Shao et al, 2019;Ozcelik et al, 2020;Gastineau et al, 2022). ViT is recently introduced into pansharpening due to its ability in capturing long-range information (Zhou et al, 2021b(Zhou et al, , 2022a.…”