PanFormer: A Transformer Based Model for Pan-Sharpening

Zhou, Huanyu; Liu, Qingjie; Wang, Yunlong

doi:10.1109/icme52920.2022.9859770

Cited by 14 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HyperTransformer can extract cross-modality dependencies in a global manner, which is completely different to convolution. Moreover, Fusformer [31] and Panformer [32] have further verified the performance of transformer architecture for pansharpening. While several Transformerbased HS pansharpening methods have been explored, challenges persist (Table 2):…”

mentioning

confidence: 73%

AIDB-Net: An Attention-Interactive Dual-Branch Convolutional Neural Network for Hyperspectral Pansharpening

Sun,

Pan

2024

Remote Sensing

View full text Add to dashboard Cite

Despite notable advancements achieved on Hyperspectral (HS) pansharpening tasks through deep learning techniques, previous methods are inherently constrained by convolution or self-attention intrinsic defects, leading to limited performance. In this paper, we proposed an Attention-Interactive Dual-Branch Convolutional Neural Network (AIDB-Net) for HS pansharpening. Our model purely consists of convolutional layers and simultaneously inherits the strengths of both convolution and self-attention, especially the modeling of short- and long-range dependencies. Specially, we first extract, tokenize, and align the hyperspectral image (HSI) and panchromatic image (PAN) by Overlapping Patch Embedding Blocks. Then, we specialize a novel Spectral-Spatial Interactive Attention which is able to globally interact and fuse the cross-modality features. The resultant token-global similarity scores can guide the refinement and renewal of the textural details and spectral characteristics within HSI features. By deeply combined these two paradigms, our AIDB-Net significantly improve the pansharpening performance. Moreover, with the acceleration by the convolution inductive bias, our interactive attention can be trained without large scale dataset and achieves competitive time cost with its counterparts. Compared with the state-of-the-art methods, our AIDB-Net makes 5.2%, 3.1%, and 2.2% improvement on PSNR metric on three public datasets, respectively. Comprehensive experiments quantitatively and qualitatively demonstrate the effectiveness and superiority of our AIDB-Net.

show abstract

mentioning

confidence: 73%

AIDB-Net: An Attention-Interactive Dual-Branch Convolutional Neural Network for Hyperspectral Pansharpening

Sun,

Pan

2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…We select nine comparative methods, including six SOTA deep learning‐based methods. Among them, PNN [14] and MSDCNN [16] are CNN‐based methods, Pan‐GAN [18] and UCGAN [19] are GAN‐based methods, and PanFormer [34] and DR‐Net [35] are transformer‐based methods. For traditional methods, we select IHS [5], SFIM [10], and BDSD [54], which represent the CS‐based method, the MRA‐based method, and the VO‐based method, respectively.…”

Section: Methodsmentioning

confidence: 99%

“…In the context of pansharpening, the use of transformers is relatively new, several recent methods have emerged that leverage transformer‐based architectures to improve the quality of pan‐sharpened images. Panformer [34] employs a transformer architecture to learn the complex relationships between low‐resolution multispectral images and high‐resolution panchromatic images. By utilizing the self‐attention mechanism, Panformer is capable of effectively capturing long‐range dependencies, leading to better spatial and spectral preservation in pan‐sharpened images.…”

Section: Related Workmentioning

confidence: 99%

SWPanGAN: A hybrid generative adversarial network for pansharpening

Huang,

Li,

Zhang

2024

IET Image Processing

View full text Add to dashboard Cite

Pansharpening is a vital technique in remote sensing that combines a low‐resolution multi‐spectral image with its corresponding panchromatic image to obtain a high‐resolution multi‐spectral image. Despite its potential benefits, the challenge lies in extracting features from the source images and eliminating artefacts in the fused images. In response to the challenge, a hybrid generative adversarial network‐based model, termed SWPanGAN, is proposed. For better feature extraction, the conventional convolution neural network is replaced with a Swin transformer in the generator, which provides the generator with the ability to model long‐range dependencies. Additionally, to suppress artefacts, a wavelet‐based discriminator is proposed for effectively distinguishing the frequency discrepancy. With these modifications, both the generator and discriminator networks of SWPanGAN are enhanced. Extensive experiments illustrate that our SWPanGAN can generate high‐quality pansharpening images and surpass other state‐of‐the‐art methods.

show abstract

“…Like the proposed CSFNet, these two methods are trained on reduced and full resolution datasets, respectively. For the supervised methods, Pannet [17], TFNet [44], and PanFormer [45] are selected to compare the fusion performance. For these three methods, due to the requirement of the reference image to supervise the training process, we only train them on the reduced resolution dataset.…”

Section: Comparison Methods and Quality Measures Metricsmentioning

confidence: 99%

Cross Spectral and Spatial Scale Non-local Attention-Based Unsupervised Pansharpening Network

Tian

Wang

et al. 2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Pansharpening means fusing the low spatial resolution multispectral image (LRMSI) and the panchromatic (PAN) image to get the high resolution multispectral image (HRMSI). Due to the powerful feature learning ability of the deep-learning (DL), DL-based unsupervised fusion methods have been developed explosively. However, most of the fusion methods are difficult to fully explore and utilize the correct spatial and spectral correlation between the LRMSI, HRMSI, and PAN images. In addition, the CNN-dominated fusion framework is limited by its local feature learning without exploring the global feature dependency to further enhance the image feature. Therefore, to fully exploit the correct correlations between LRMSI, HRMSI, and PAN images and to explore the global feature dependency, we designed a cross-scale unsupervised fusion network (CSFNet). This network is composed of two cross spectral and spatial scale's nonlocal attention blocks to effectively fuse the LRMSI and PAN image features. And the fusion strategy is implemented by mapping the computed nonlocal similarity from the low resolution scale to the high resolution scale and outputs the reconstructed HRMSI feature. The experimental results on two datasets show that it achieves state-of-the-art performance compared to other fusion methods.

show abstract

PanFormer: A Transformer Based Model for Pan-Sharpening

Cited by 14 publications

References 16 publications

AIDB-Net: An Attention-Interactive Dual-Branch Convolutional Neural Network for Hyperspectral Pansharpening

AIDB-Net: An Attention-Interactive Dual-Branch Convolutional Neural Network for Hyperspectral Pansharpening

SWPanGAN: A hybrid generative adversarial network for pansharpening

Cross Spectral and Spatial Scale Non-local Attention-Based Unsupervised Pansharpening Network

Contact Info

Product

Resources

About