Hyperspectral Image Super-Resolution with Self-Supervised Spectral-Spatial Residual Network

Abstract

Recently, many convolutional networks have been built to fuse a low spatial resolution (LR) hyperspectral image (HSI) and a high spatial resolution (HR) multispectral image (MSI) to obtain HR HSIs. However, most deep learning-based methods are supervised methods, which require sufficient HR HSIs for supervised training. Collecting plenty of HR HSIs is laborious and time-consuming. In this paper, a self-supervised spectral-spatial residual network (SSRN) is proposed to alleviate dependence on a mass of HR HSIs. In SSRN, the fusion of HR MSIs and LR HSIs is considered a pixel-wise spectral mapping problem. Firstly, this paper assumes that the spectral mapping between HR MSIs and HR HSIs can be approximated by the spectral mapping between LR MSIs (derived from HR MSIs) and LR HSIs. Secondly, the spectral mapping between LR MSIs and LR HSIs is explored by SSRN. Finally, a self-supervised fine-tuning strategy is proposed to transfer the learned spectral mapping to generate HR HSIs. SSRN does not require HR HSIs as the supervised information in training. Simulated and real hyperspectral databases are utilized to verify the performance of SSRN.

Deep-Learning-Based Multispectral Image Reconstruction from Single Natural Color RGB Image—Enhancing UAV-Based Phenotyping

“…However, the final quality of generated HR-MSIs should not be affected if the super-resolution scale is low, e.g., only 4 in our study. Another advantage of direct reconstruction methods in MSI super-resolution is that the HR-MSIs are reconstructed directly based on the HR-RGB image, taking full advantage of the spatial information of HR-RGB and leading to marginal spatial distortion in the derived HR-MSIs [57]. Additionally, the fusion strategy normally requires a pair of existing low-resolution spectral images and HR-RGB-largely limiting the transferability during practical applications.…”

Self-Supervised Learning in Remote Sensing: A review

“…Remote sensing. Autoencoders have been widely used to learn representation from various remote sensing data like multispectral images [92,93,94,95,96,97,98,99], hyperspectral images [100,101,102,103,104,105,106,107] and SAR images [108,109,110,111]. Lu et al [92] proposed a combination of a shallowly weighted de-convolution network with a spatial pyramid model in order to learn multi-layer feature maps and filters for input images.…”

Hyperspectral Image Super-Resolution with RGB Image Super-Resolution as an Auxiliary Task

“…This trend highlights the need for bigger training datasets. Due to the challenge of reconstructing HSI from RGBIs that contain only three bands, there emerges research applying MSIs containing 3-8 bands to reconstruct HSIs [14].…”