Hyperspectral Image Super Resolution Based on Multiscale Feature Fusion and Aggregation Network With 3-D Convolution

Abstract

The spectral resolution of hyperspectral images (HSIs) is very high. Nevertheless, their spatial resolution is low due to various hardware limitations. Therefore, it is important to study HSI super-resolution to improve their spatial resolution. In this paper, for hyperspectral single-image super-resolution, we propose a multi-scale feature fusion and aggregation network with 3D convolution (MFFA-3D) by cascading the MFFA-3D block. The MFFA-3D block includes group multi-scale feature fusion part and multi-scale feature aggregation part. In group multi-scale feature fusion part, a novel group multi-scale feature fusion method is proposed. Group feature fusion module and two-step multi-scale module are proposed in multi-scale feature aggregation part. In order to prevent spectral distortion, a spectral gradient loss function is proposed and combined with the mean square error (MSE) loss function to form the final loss function. Since the proposed super-resolution network is full 3D convolutional network, our method can perform direct super-resolution transfer even if the number of the bands of test images is different from that of training images. The experiments over simulated and real hyperspectral images demonstrate the superiority of the proposed method in terms of qualitative and quantitative evaluation.

Bidirectional 3D Quasi-Recurrent Neural Network for Hyperspectral Image Super-Resolution

“…Therefore, they proposed a mixed convolutional network (MCNet) for HSI super-resolution. Hu et al [22] designed a multiple feature fusion and aggregation network with 3D convolution (MFFA-3D) equipped with multi-scale connections and two-step multi-scale strategy to obtain the high-resolution HSI. These 3D deep learning-based methods have achieved effective reconstruction results.…”

“…Previous works [9], [12], [22], [42] have modeled HSIs with 3D CNN, which significantly improve the quality of reconstructed HSIs and make model itself more flexible on the number of bands. Besides, 3D CNN can well model the spatial-spectral correlation.…”

“…Recently, convolutional neural network (CNN) based HSI super-resolution methods are presented [14], [19]- [22]. Yuan et al [23] and Dong et al [24] extend RGB based superresolution networks for HSI super-resolution, but these methods usually lack the capability to extract important information from different spectral channels, because they are mostly implemented with 2D CNNs.…”

Hyperspectral Image Super-Resolution Based on Spatial Correlation-Regularized Unmixing Convolutional Neural Network

“…The experiments demonstrate its higher quality both in reconstruction and spectral fidelity. Arun and Hu et al [34,35] also proposed two additional 3D CNNbased SR methods. Arun et al [34] utilized a convolution-deconvolution framework and hypercube-specific loss functions.…”

“…Moreover, the spatial-spectral accuracy of the superresolved hypercubes, in terms of the validity of regularizing features and endmembers, was explored to devise an optimal ensemble strategy. Hu et al [35] utilized a multiscale feature fusion and aggregation network with 3D convolution, and proposed a spectral gradient loss function to prevent spectral distortion. Inspired by the unmixing idea of [6], a deep feature matrix factorization (DFMF)-based SR method was proposed in [36], by incorporating a CNN and nonnegative matrix factorization strategy.…”