Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network

Abstract

In this paper, we propose a method using a three dimensional convolutional neural network (3-D-CNN) to fuse together multispectral (MS) and hyperspectral (HS) images to obtain a high resolution hyperspectral image. Dimensionality reduction of the hyperspectral image is performed prior to fusion in order to significantly reduce the computational time and make the method more robust to noise. Experiments are performed on a data set simulated using a real hyperspectral image. The results obtained show that the proposed approach is very promising when compared to conventional methods. This is especially true when the hyperspectral image is corrupted by additive noise.

Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network

“…For instance, the accuracy of MS and HS image fusion can also be improved by using three-dimensional CNN [18], in which the extremely high dimensionality of hyper-spectral images are simply reduced by principle component analysis (PCA), while a recent study [19] has indicated that saliency-based band selection can be studied to compress and better represent the rich spectral information. Thus, as our future works tend to process hyper-spectral data (Quality restoration, fusion, interpretation), the combination of feature learning based on neurocomputing and saliency detection based on manifold feature representation [20][21] shall be focused in particular.…”

Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net

“…These methods can be easily adapted to MS/HS fusion problem. For example, very recently, [28] proposed a 3D-CNN based MS/HS fusion method by using PCA to reduce the computational cost. This method is usually trained with prepared training data.…”

“…As compared with conventional methods, these DL based ones are superior in that they need fewer assumptions on the prior knowledge of the to-be-recovered HrHS, while can be directly trained on a set of paired training data simulating the network inputs (LrHS&HrMS images) and outputs (HrHS images). The most commonly employed network structures include CNN [7], 3D CNN [28], and residual net [30]. Like other image restoration tasks where DL is successfully applied to, these DL-based methods have also achieved good resolution performance for MS/MS fusion task.…”

A CNN-Based Fusion Method for Feature Extraction from Sentinel Data

“…According to the taxonomy given in [5] data fusion methods, i.e., processing dealing with data and information from multiple sources to achieve improved information for decision making can be grouped into three main categories: -pixel-level: the pixel values of the sources to be fused are jointly processed [6][7][8][9]; -feature-level: features like lines, regions, keypoints, maps, and so on, are first extracted independently from each source image and subsequently combined to produce higher-level cross-source features, which may represent the desired output or be further processed [10][11][12][13][14][15][16][17]; -decision-level: the high-level information extracted independently from each source is combined to provide the final outcome, for example using fuzzy logic [18,19], decision trees [20], Bayesian inference [21], Dempster-Shafer theory [22], and so forth.…”

“…-mixed: the above cases may also occur jointly, generating mixed situations. For example, hyperspectral and multiresolution images can be fused to produce a spatial-spectral full-resolution datacube [9,42]. Likewise, low-resolution temporally-dense series can be fused with high-resolution, but temporally sparse ones to simulate a temporal-spatial full-resolution sequence [43].…”