A New Variational Approach Based on Proximal Deep Injection and Gradient Intensity Similarity for Spatio-Spectral Image Fusion

Abstract

Pansharpening is a very debated spatio-spectral fusion problem. It refers to the fusion of a high spatial resolution panchromatic (PAN) image with a lower spatial but higher spectral resolution multispectral (LRMS) image in order to obtain an image with high resolution in both the domains. In this paper, we propose a novel variational optimizationbased (VO) approach to address this issue incorporating the outcome of a deep convolutional neural network (DCNN). This solution can take advantages of both the paradigms. On one hand, higher performance can be expected introducing machine learning methods based on the training by examples philosophy into VO approaches. On other hand, the combination of VO techniques with DCNNs can aid the generalization ability of these latter. In particular, we formulate a 2-based proximal deep injection term to evaluate the distance between the DCNN outcome and the desired high spatial resolution multispectral image. This represents the regularization term for our VO model. Furthermore, a new data fitting term measuring the spatial fidelity is proposed. Finally, the proposed convex VO problem is efficiently solved by exploiting the framework of the alternating direction method of multipliers, thus guaranteeing the convergence of the algorithm. Extensive experiments both on simulated and real datasets demonstrate that the proposed approach can outperform state-of-the-art spatio-spectral fusion methods, even showing a significant generalization ability. Please find the project page: https://liangjiandeng.github.io/Projects Res/DMPIF 2020jstars.html.

Maskpan: Mask Prior Guided Network For Pansharpening

“…[17] proposes a representation between HRMS and PAN by linear combinations of multiorder gradients. [19] improves the spectral term proposed in [18] by adjusting the mean of the extended PAN. Besides, [19] also proposes to use the output of a pre-trained neural network to guided the restoration of HRMS.…”

“…[19] improves the spectral term proposed in [18] by adjusting the mean of the extended PAN. Besides, [19] also proposes to use the output of a pre-trained neural network to guided the restoration of HRMS. [20] proposes to apply the MRA-based model to formulate the relationship between HRMS and PAN, which achieves remarkable performance.…”

Remote Sensing Pansharpening by Full-Depth Feature Fusion

“…Recently, with the rapid development of deep learning and accessibility of highperformance computing hardware equipment, convolutional neural networks (CNNs) have shown outstanding performance in image processing fields, e.g., image resolution reconstruction [45][46][47][48][49], image segmentation [50][51][52], image fusion [53][54][55][56][57], image classification [58], image denoising [59], etc. Therefore, many methods [34][35][36][37][38]41,42,[58][59][60][61][62][63][64][65][66][67][68][69][70][71][72][73][74][75] based on deep learning have also been applied to solve the pansharpening problem.…”