Pansharpening by Convolutional Neural Networks

Abstract

Abstract:A new pansharpening method is proposed, based on convolutional neural networks. We adapt a simple and effective three-layer architecture recently proposed for super-resolution to the pansharpening problem. Moreover, to improve performance without increasing complexity, we augment the input by including several maps of nonlinear radiometric indices typical of remote sensing. Experiments on three representative datasets show the proposed method to provide very promising results, largely competitive with the current state of the art in terms of both full-reference and no-reference metrics, and also at a visual inspection.

VPRS-Based Regional Decision Fusion of CNN and MRF Classifications for Very Fine Resolution Remotely Sensed Images

“…Spatial features in remotely sensed data like VFSR imagery are intrinsically local and stationary that represent a coherent spatial pattern [58]. The presence of such spatial features are detected by the convolutional filters within the CNN, and well generalized into increasingly abstract and robust features through hierarchical feature representations.…”

A CNN-Based Fusion Method for Feature Extraction from Sentinel Data

“…In the last few years, CNNs have been successfully applied to many classical image processing problems, such as denoising [50], super-resolution [51], pansharpening [8,24], segmentation [52], object detection [53,54], change detection [27] and classification [17,[55][56][57]. The main strengths of CNNs are (i) an extreme versatility that allows them to approximate any sort of linear or non-linear transformation, including scaling or hard thresholding; (ii) no need to design handcrafted filters, replaced by machine learning; (iii) high-speed processing, thanks to parallel computing.…”

“…According to the taxonomy given in [5] data fusion methods, i.e., processing dealing with data and information from multiple sources to achieve improved information for decision making can be grouped into three main categories: -pixel-level: the pixel values of the sources to be fused are jointly processed [6][7][8][9]; -feature-level: features like lines, regions, keypoints, maps, and so on, are first extracted independently from each source image and subsequently combined to produce higher-level cross-source features, which may represent the desired output or be further processed [10][11][12][13][14][15][16][17]; -decision-level: the high-level information extracted independently from each source is combined to provide the final outcome, for example using fuzzy logic [18,19], decision trees [20], Bayesian inference [21], Dempster-Shafer theory [22], and so forth.…”

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

“…The means to fuse HSI with these auxiliary data in the deep learning framework still lacks study. [38] residual CNN; spectral regularizer is used in loss function SDCNN [39] CNN to learn the spectral difference PNN [40] CNN for pan-sharpening MSI pan-sharpening DRPNN [41] residual CNN for pan-sharpening PanNet [42] residual CNN; learn mapping in high-frequency domain MSDCNN [43] two CNN branches with different depths; multi-scale kernels in each convolutional layer…”

“…CNN has also been applied to pan-sharpening. In [40], HR panchromatic image was stacked with up-scaled LR MSI to form an input cube, a pan-sharpening CNN network (PNN) was used to learn the mapping between the input cube and HR MSI. A deep residual PNN (DRPNN) model was proposed to boost PNN by using residual learning in [41].…”