Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening

Deng, Liang-Jian; Vivone, Gemine; Jin, Cheng; Chanussot, Jocelyn

doi:10.1109/tgrs.2020.3031366

Cited by 168 publications

(89 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some approaches enhance the ability to fuse images in the network structures, such as 3D convolutional neural networks (CNN) [38], residual networks [39], multiscale structures [40], pyramid networks [41], attention networks [42], [43], crossmode information [44], dense networks [45], [46], adversarial network [47], [48]. Others use detail information from highspatial-resolution conventional images to improve performance [49]- [52], while some form a hybrid of model-and deep learning-based approaches [53]- [55], [65], [66].…”

Section: B Deep Learning-based Approachesmentioning

confidence: 99%

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Liu

Shen

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Hyperspectral and multispectral image fusion aims to fuse a low-spatial-resolution hyperspectral image (HSI) and a high-spatial-resolution multispectral image to form a highspatial-resolution HSI. Motivated by the success of model-and deep learning-based approaches, we propose a novel patch-aware deep fusion approach for HSI by unfolding a subspace-based optimization model, where moderate-sized patches are used in both training and test phases. The goal of this approach is to make full use of the information of patch under subspace representation, restrict the scale and enhance the interpretability of the deep network, thereby improving the fusion. First, a subspacebased fusion model was built with two regularization terms to localize pixels and extract texture. Then, the subspace-based fusion model was solved by the alternating direction method of multipliers algorithm, and the model was divided into one fidelitybased problem and two regularization-based problems. Finally, a structured deep fusion network was proposed by unfolding all steps of the algorithm as network layers. Specifically, the fidelity-based problem was solved by a gradient descent algorithm and implemented by a network. The two regularization-based problems were described by proximal operators and learnt by two u-shaped architectures. Moreover, an aggregation fusion technique was proposed to improve the performance by averaging the fused images in all iterations and aggregating the overlapping patches in the test phase. Experimental results, conducted on both synthetic and real datasets, demonstrated the effectiveness of the proposed approach.

show abstract

Section: B Deep Learning-based Approachesmentioning

confidence: 99%

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Liu

Shen

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…He et al [38] designed different convolution neural networks (CNNs) to extract proper details for injection. Deng et al [39] also developed CNNs for pansharpening, where the difference images between PAN and LR MS images are fed into the networks. Guo et al [40] combined the multiscale recursive blocks and the anisotropic total variation to overcome the spatial distortions.…”

Section: Multispectral (Hr Ms) Imagesmentioning

confidence: 99%

Exploiting Low-Rank and Sparse Properties in Strided Convolution Matrix for Pansharpening

Zhang

et al. 2021

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Fusion of low spatial resolution multispectral (LR MS) and panchromatic (PAN) images to acquire high spatial resolution multispectral (HR MS) images has attracted increasing attention in recent years. In this paper, we first utilize the form of convolution matrix (CM) to formulate the image fusion problem. In order to reduce the complexity of CM, the step size is introduced and strided convolution matrix (SCM) is constructed. Then, we explore the low-rank property in SCM and impose the prior on the spatial and spectral degradation model of LR MS and PAN images. Meanwhile, sparsity in SCM is considered to further enhance the local structures in the fused image. Finally, the proposed model is optimized efficiently by the alternative direction method of multipliers (ADMM). By exploiting the low-rank and sparse priors in SCM of HR MS image, the local and global structures can be better preserved. The experimental results on the reduced-resolution and full-resolution datasets also show that the proposed method behaves well in qualitative and quantitative assessments.

show abstract

“…Over the past few decades, a large variety of pansharpening methods have been proposed. Most of them can be divided into the following four categories: Component substitution (CS) [2][3][4][5] approaches, multi-resolution analysis (MRA) [1,[6][7][8][9][10][11] approaches, variational optimization (VO) approaches, and deep learning (DL) approaches [34][35][36][37][38][39][40][41][42].…”

Section: Introductionmentioning

confidence: 99%

“…Recently, with the rapid development of deep learning and accessibility of highperformance computing hardware equipment, convolutional neural networks (CNNs) have shown outstanding performance in image processing fields, e.g., image resolution reconstruction [45][46][47][48][49], image segmentation [50][51][52], image fusion [53][54][55][56][57], image classification [58], image denoising [59], etc. Therefore, many methods [34][35][36][37][38]41,42,[58][59][60][61][62][63][64][65][66][67][68][69][70][71][72][73][74][75] based on deep learning have also been applied to solve the pansharpening problem. Benefiting from the powerful nonlinear fitting and feature extraction capabilities of CNNs and the availability of big data, these DL-based methods could perform better than the above three methods to a certain degree, i.e., CS-, MRA-, and VO-based methods.…”

Section: Introductionmentioning

confidence: 99%

Remote Sensing Pansharpening by Full-Depth Feature Fusion

et al. 2022

Self Cite

View full text Add to dashboard Cite

Pansharpening is an important yet challenging remote sensing image processing task, which aims to reconstruct a high-resolution (HR) multispectral (MS) image by fusing a HR panchromatic (PAN) image and a low-resolution (LR) MS image. Though deep learning (DL)-based pansharpening methods have achieved encouraging performance, they are infeasible to fully utilize the deep semantic features and shallow contextual features in the process of feature fusion for a HR-PAN image and LR-MS image. In this paper, we propose an efficient full-depth feature fusion network (FDFNet) for remote sensing pansharpening. Specifically, we design three distinctive branches called PAN-branch, MS-branch, and fusion-branch, respectively. The features extracted from the PAN and MS branches will be progressively injected into the fusion branch at every different depth to make the information fusion more broad and comprehensive. With this structure, the low-level contextual features and high-level semantic features can be characterized and integrated adequately. Extensive experiments on reduced- and full-resolution datasets acquired from WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate that the proposed FDFNet only with less than 100,000 parameters performs better than other detail injection-based proposals and several state-of-the-art approaches, both visually and quantitatively.

show abstract

Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening

Cited by 168 publications

References 59 publications

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Exploiting Low-Rank and Sparse Properties in Strided Convolution Matrix for Pansharpening

Remote Sensing Pansharpening by Full-Depth Feature Fusion

Contact Info

Product

Resources

About