Hyperspectral Pansharpening Based on Spectral Constrained Adversarial Autoencoder

Abstract

Hyperspectral (HS) imaging is conducive to better describing and understanding the subtle differences in spectral characteristics of different materials due to sufficient spectral information compared with traditional imaging systems. However, it is still challenging to obtain high resolution (HR) HS images in both the spectral and spatial domains. Different from previous methods, we first propose spectral constrained adversarial autoencoder (SCAAE) to extract deep features of HS images and combine with the panchromatic (PAN) image to competently represent the spatial information of HR HS images, which is more comprehensive and representative. In particular, based on the adversarial autoencoder (AAE) network, the SCAAE network is built with the added spectral constraint in the loss function so that spectral consistency and a higher quality of spatial information enhancement can be ensured. Then, an adaptive fusion approach with a simple feature selection rule is induced to make full use of the spatial information contained in both the HS image and PAN image. Specifically, the spatial information from two different sensors is introduced into a convex optimization equation to obtain the fusion proportion of the two parts and estimate the generated HR HS image. By analyzing the results from the experiments executed on the tested data sets through different methods, it can be found that, in CC, SAM, and RMSE, the performance of the proposed algorithm is improved by about 1.42%, 13.12%, and 29.26% respectively on average which is preferable to the well-performed method HySure. Compared to the MRA-based method, the improvement of the proposed method in in the above three indexes is 17.63%, 0.83%, and 11.02%, respectively. Moreover, the results are 0.87%, 22.11%, and 20.66%, respectively, better than the PCA-based method, which fully illustrated the superiority of the proposed method in spatial information preservation. All the experimental results demonstrate that the proposed method is superior to the state-of-the-art fusion methods in terms of subjective and objective evaluations.

Spectral-Fidelity Convolutional Neural Networks for Hyperspectral Pansharpening

“…Over recent years, convolutional neural networks (CNNs) have experienced significant success in tasks related to the enhancement of spatial resolution [26]- [36]. Dong et al originally proposed a three-layer super-resolution CNN (SRCNN) to learn the mapping between LR and HR images [26].…”

“…Xie et al treated hyperspectral pansharpening as a constrained minimization problem with additional priors learned by CNN [35]. He et al proposed spectral constrained adversarial autoencoder to extract deep features of HS images, designed an adaptive fusion approach ruled by feature selection and estimated high-resolution HS images through solving an optimization equation [36].…”

Generative Adversarial Network for Pansharpening With Spectral and Spatial Discriminators

“…In our RDGAN approach, we consider a Residual Dense architecture for the generator with a geometrical constraint in the non adversarial loss function to restore the spatial resolution of images. He et al [19] preserve the spectral resolution by adding a regularisation term based on the Spectral Angle Map (SAM) measure in the generator in an adversarial autoencoder framework. This measure computes the spectral distortion between two images and they propose to minimize this quantity in the loss function.…”

Unsupervised Deep Learning for Landslide Detection from Multispectral Sentinel-2 Imagery

“…Therefore, some researchers have employed AEs to solve critical image processing challenges such as image classification [24][25][26], clustering [23,27,28], spectral unmixing [29] and image segmentation [30,31]. AEs have also been applied to deal with other important problems such as image fusion [32], change detection [33][34][35], pansharpening [36,37], anomaly detection [38,39], and image retrieval [40]. In recent years, the application of AE for clustering purposes has gained much attention in the RS community.…”