Hyperspectral Image Super-Resolution with RGB Image Super-Resolution as an Auxiliary Task

Li, Ké; Dai, Dengxin; Gool, Luc Van

doi:10.1109/wacv51458.2022.00409

Cited by 13 publications

(14 citation statements)

References 94 publications

(241 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover we conduct an ablation experiment (Sec. IV-C) that proves that pretraining with our synthetic dataset leads to better performance than using RGB images as an auxiliary task [7]. We also would like to remark that our results raise questions about the significance of results assessing merits of neural network design obtained on small datasets.…”

Section: Introductionmentioning

confidence: 67%

“…Most of the current work focuses on the design of novel neural network architectures, potentially exploiting clever priors or layer structures in their operations. On the other hand, it is well known [6], [7] that training on more data is often more impactful than revising architectural design. Moreover, using small datasets, such as the ones in the current literature, poses the risk of producing unreliable scientific results when assessing the merits of a design over another.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Synthetic Data Pretraining for Hyperspectral Image Super-Resolution

Aiello,

Agarla,

Valsesia

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Large-scale self-supervised pretraining of deep learning models is known to be critical in several fields, such as language processing, where its has led to significant breakthroughs. Indeed, it is often more impactful than architectural designs. However, the use of self-supervised pretraining lags behind in several domains, such as hyperspectral images, due to data scarcity. This paper addresses the challenge of data scarcity in the development of methods for spatial super-resolution of hyperspectral images (HSI-SR). We show that state-of-the-art HSI-SR methods are severely bottlenecked by the small paired datasets that are publicly available, also leading to unreliable assessment of the architectural merits of the models. We propose to capitalize on the abundance of high resolution (HR) RGB images to develop a self-supervised pretraining approach that significantly improves the quality of HSI-SR models. In particular, we leverage advances in spectral reconstruction methods to create a vast dataset with high spatial resolution and plausible spectra from RGB images, to be used for pretraining HSI-SR methods. Experimental results, conducted across multiple datasets, report large gains for state-of-the-art HSI-SR methods when pretrained according to the proposed procedure, and also highlight the unreliability of ranking methods when training on small datasets.

show abstract

Section: Introductionmentioning

confidence: 67%

Section: Introductionmentioning

confidence: 99%

Synthetic Data Pretraining for Hyperspectral Image Super-Resolution

Aiello,

Agarla,

Valsesia

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Additionally, the authors also devise a novel augmentation algorithm called Spectral Mixup to increase the amount of training samples. The most recent version of this approach was published by the same authors in 2022 [70]. This method outperforms [43], [44], [46], and [42].…”

Section: B Deep Learning-based Sisrmentioning

confidence: 99%

“…In [44] and [45], the scaling factor goes up to ×8 at most, and HSI-SR requires higher scaling factors in order to be put into practical use. Furthermore, SISR techniques have no unsupervised approaches associated with DCNNs, as the only semi-supervised approach that tackles data scarcity problem is the one mentioned in [69], [70].…”

Section: B Deep Learning-based Sisrmentioning

confidence: 99%

A Review of Spatial Enhancement of Hyperspectral Remote Sensing Imaging Techniques

Aburaed

Alkhatib

Marshall

et al. 2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Remote sensing technology has undeniable importance in various industrial applications, such as mineral exploration, plant detection, defect detection in aerospace and shipbuilding, and optical gas imaging, to name a few. Remote sensing technology has been continuously evolving, offering a range of image modalities that can facilitate the aforementioned applications. One such modality is Hyperspectral Imaging (HSI). Unlike Multispectral Images (MSI) and natural images, HSI consist of hundreds of bands. Despite their high spectral resolution, HSI suffer from low spatial resolution in comparison to their MSI counterpart, which hinders the utilization of their full potential. Therefore, spatial enhancement, or Super Resolution (SR), of HSI is a classical problem that has been gaining rapid attention over the past two decades. The literature is rich with various SR algorithms that enhance the spatial resolution of HSI while preserving their spectral fidelity. This paper reviews and discusses the most important algorithms relevant to this area of research between 2002-2022, along with the most frequently used datasets, HSI sensors, and quality metrics. Meta-analysis are drawn based on the aforementioned information, which is used as a foundation that summarizes the state of the field in a way that bridges the past and the present, identifies the current gap in it, and recommends possible future directions.

show abstract

“…To address this ill-posed problem, many SSR approaches have been proposed, which can be divided into two categories: shallow learning and deep learning [14]. Early researchers mainly used sparse coding or relatively shallow learning methods to explore the prior information of HSIs.…”

Section: Introductionmentioning

confidence: 99%

LSTNet: A Reference-Based Learning Spectral Transformer Network for Spectral Super-Resolution

Yuan

Jiang

et al. 2022

Sensors

View full text Add to dashboard Cite

Hyperspectral images (HSIs) are data cubes containing rich spectral information, making them beneficial to many Earth observation missions. However, due to the limitations of the associated imaging systems and their sensors, such as the swath width and revisit period, hyperspectral imagery over a large coverage area cannot be acquired in a short amount of time. Spectral super-resolution (SSR) is a method that involves learning the relationship between a multispectral image (MSI) and an HSI, based on the overlap region, followed by reconstruction of the HSI by making full use of the large swath width of the MSI, thereby improving its coverage. Much research has been conducted recently to address this issue, but most existing methods mainly learn the prior spectral information from training data, lacking constraints on the resulting spectral fidelity. To address this problem, a novel learning spectral transformer network (LSTNet) is proposed in this paper, utilizing a reference-based learning strategy to transfer the spectral structure knowledge of a reference HSI to create a reasonable reconstruction spectrum. More specifically, a spectral transformer module (STM) and a spectral reconstruction module (SRM) are designed, in order to exploit the prior and reference spectral information. Experimental results demonstrate that the proposed method has the ability to produce high-fidelity reconstructed spectra.

show abstract

Hyperspectral Image Super-Resolution with RGB Image Super-Resolution as an Auxiliary Task

Cited by 13 publications

References 94 publications

Synthetic Data Pretraining for Hyperspectral Image Super-Resolution

Synthetic Data Pretraining for Hyperspectral Image Super-Resolution

A Review of Spatial Enhancement of Hyperspectral Remote Sensing Imaging Techniques

LSTNet: A Reference-Based Learning Spectral Transformer Network for Spectral Super-Resolution

Contact Info

Product

Resources

About