Machine Learning in Pansharpening: A benchmark, from shallow to deep networks

Deng, Liang-Jian; Vivone, Gemine; Paoletti, Mercedes E.; Scarpa, Giuseppe; He, Jiang; Zhang, Yongjun; Chanussot, Jocelyn

doi:10.1109/mgrs.2022.3187652

Cited by 67 publications

(4 citation statements)

References 117 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They are listed in Tab.IV, grouped by their general approach: component substitution, multiresolution analysis, variational optimization, machine learning, the latter trained at reduced or full resolution. Most of the methods are available in the toolboxes [2] and [54], from which we selected those that performed best in the experiments. Methods of the last group, instead, were downloaded from the authors' websites.…”

Section: B Reference Methodsmentioning

confidence: 99%

“…In this Section, we review the state of the art on deep learning-based pansharpening. However, we neglect methods trained at reduced resolution, referring the reader to a recent review [54], and focus on those performing unsupervised training in the full resolution domain, more strictly related to our proposal. In addition, we describe the correlation-based spatial loss proposed in [50] and adopted here.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Fast Full-Resolution Target-Adaptive CNN-Based Pansharpening Framework

Ciotola

Scarpa

2023

Remote Sensing

Self Cite

View full text Add to dashboard Cite

In the last few years, there has been a renewed interest in data fusion techniques, and, in particular, in pansharpening due to a paradigm shift from model-based to data-driven approaches, supported by the recent advances in deep learning. Although a plethora of convolutional neural networks (CNN) for pansharpening have been devised, some fundamental issues still wait for answers. Among these, cross-scale and cross-datasets generalization capabilities are probably the most urgent ones since most of the current networks are trained at a different scale (reduced-resolution), and, in general, they are well-fitted on some datasets but fail on others. A recent attempt to address both these issues leverages on a target-adaptive inference scheme operating with a suitable full-resolution loss. On the downside, such an approach pays an additional computational overhead due to the adaptation phase. In this work, we propose a variant of this method with an effective target-adaptation scheme that allows for the reduction in inference time by a factor of ten, on average, without accuracy loss. A wide set of experiments carried out on three different datasets, GeoEye-1, WorldView-2 and WorldView-3, prove the computational gain obtained while keeping top accuracy scores compared to state-of-the-art methods, both model-based and deep-learning ones. The generality of the proposed solution has also been validated, applying the new adaptation framework to different CNN models.

show abstract

Section: B Reference Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Fast Full-Resolution Target-Adaptive CNN-Based Pansharpening Framework

Ciotola

Scarpa

2023

Remote Sensing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to the decent performance, pan-sharpening has attracted a great deal of attention and many pan-sharpening methods have been proposed [8]- [9], which can be mainly classified into four groups: 1) component substitution (CS) based methods, 2) multiresolution analysis (MRA) based methods, 3) degradation model (DM) based methods, and 4) DL-based methods. They are introduced successively in Sections II.A-II.D.…”

Section: Related Workmentioning

confidence: 99%

A Unified Two-Stage Spatial and Spectral Network With Few-Shot Learning for Pansharpening

Sheng

Zhang

Sun

et al. 2023

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

Recently, pan-sharpening methods based on deep learning (DL) have achieved state-of-the-art results. However, current existing DL-based pan-sharpening methods need to be trained repetitively for different satellite sensors to obtain satisfactory fusion performance and therefore require a large number of training images for each satellite. To deal with these issues, in this paper we propose a unified two-stage spatial and spectral network (UTSN) for pan-sharpening. A branch of networks is constructed for each different satellite, in which the spatial enhancement network (SEN) is shared to improve the spatial details in the fused images from different satellites. A spectral adjustment network (SAN) is employed to capture the spectral characteristics of the specific satellite. Through SAN, the spectral information in the intermediate image from SEN is refined to produce the final fusion results. Such a framework can integrate the datasets from different satellites together for sufficient training of SEN. The proposed method is able to achieve promising pan-sharpening results also for a new satellite with limited training images by only learning a new SAN on the few-shot datasets due to the simple but efficient structure of SAN. The experimental results show that the proposed method can produce state-of-the-art fusion results in both the standard and few-shot cases. The source code is publicly available at https://github.com/RSMagneto/UTSN.

show abstract

“…High-resolution multispectral (HRMS) images are required for most remote sensing applications. Due to physical constraints, it is impractical to acquire such images using a single sensor [1]. As an alternative, modern satellites carry two types of sensors to capture multimodal data with different yet complementary images: panchromatic (PAN) images and multispectral (MS) images.…”

Section: Introductionmentioning

confidence: 99%

Triple‐loss driven generative adversarial network for pansharpening

Huang,

Li,

Zhang

2023

IET Image Processing

View full text Add to dashboard Cite

Pansharpening aims at fusing a panchromatic (PAN) image and a low‐resolution multispectral (LRMS) image into a high‐resolution multispectral (HRMS) image. In recent years, GAN‐based pansharpening methods have achieved excellent results, but they suffer from inadequate feature preservation and unstable training. To address these issues, a novel GAN‐based model named TriLossGAN is proposed. This method constructs three loss components with the help of the generator and the dual‐discriminator, which are calculated in both the original spatial domain and the transform domain to better preserve high‐frequency and low‐frequency information in the fused image. Additionally, a new training strategy is designed to stabilize the training process. In extensive experiments, the proposed method achieved satisfactory results on three datasets with QNR values of 0.9584 on GaoFen‐2, 0.9601 on QuickBird, and 0.9138 on WorldView‐3. Qualitative and quantitative comparisons demonstrate that TriLossGAN outperforms other state‐of‐the‐art methods.

show abstract

Machine Learning in Pansharpening: A benchmark, from shallow to deep networks

Cited by 67 publications

References 117 publications

Fast Full-Resolution Target-Adaptive CNN-Based Pansharpening Framework

Fast Full-Resolution Target-Adaptive CNN-Based Pansharpening Framework

A Unified Two-Stage Spatial and Spectral Network With Few-Shot Learning for Pansharpening

Triple‐loss driven generative adversarial network for pansharpening

Contact Info

Product

Resources

About