SSR-NET: Spatial–Spectral Reconstruction Network for Hyperspectral and Multispectral Image Fusion

Zhang, Xueting; Huang, Wei; Wang, Qi; Li, Xuelong

doi:10.1109/tgrs.2020.3018732

Cited by 107 publications

(49 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Learning-based approaches often build a deep network to describe the fusion process, and produce the target image by feeding observed images into the network [11], [14], [36]. Some approaches enhance the ability to fuse images in the network structures, such as 3D convolutional neural networks (CNN) [38], residual networks [39], multiscale structures [40], pyramid networks [41], attention networks [42], [43], crossmode information [44], dense networks [45], [46], adversarial network [47], [48]. Others use detail information from highspatial-resolution conventional images to improve performance [49]- [52], while some form a hybrid of model-and deep learning-based approaches [53]- [55], [65], [66].…”

Section: B Deep Learning-based Approachesmentioning

confidence: 99%

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Liu

Shen

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Hyperspectral and multispectral image fusion aims to fuse a low-spatial-resolution hyperspectral image (HSI) and a high-spatial-resolution multispectral image to form a highspatial-resolution HSI. Motivated by the success of model-and deep learning-based approaches, we propose a novel patch-aware deep fusion approach for HSI by unfolding a subspace-based optimization model, where moderate-sized patches are used in both training and test phases. The goal of this approach is to make full use of the information of patch under subspace representation, restrict the scale and enhance the interpretability of the deep network, thereby improving the fusion. First, a subspacebased fusion model was built with two regularization terms to localize pixels and extract texture. Then, the subspace-based fusion model was solved by the alternating direction method of multipliers algorithm, and the model was divided into one fidelitybased problem and two regularization-based problems. Finally, a structured deep fusion network was proposed by unfolding all steps of the algorithm as network layers. Specifically, the fidelity-based problem was solved by a gradient descent algorithm and implemented by a network. The two regularization-based problems were described by proximal operators and learnt by two u-shaped architectures. Moreover, an aggregation fusion technique was proposed to improve the performance by averaging the fused images in all iterations and aggregating the overlapping patches in the test phase. Experimental results, conducted on both synthetic and real datasets, demonstrated the effectiveness of the proposed approach.

show abstract

Section: B Deep Learning-based Approachesmentioning

confidence: 99%

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Liu

Shen

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Despite the undisputed need for the concurrent use of multiple visualizations, there is a need for a single "best" visualization for dissemination, fieldwork, and as an input to machine learning. A recent development in this area in the field of archaeological LiDAR is image fusion, a well-known technique in remote sensing that creates in which a meaningful image combination that preserves the positive characteristics of each input image [71,[74][75][76][77][78]. However, applying image fusion to DFM visualizations can further exacerbate the shortcomings typical of complex visualizations: as the complexity grows, it becomes increasingly difficult to interpret the image without specialized training.…”

Section: Enhanced Visualizationmentioning

confidence: 99%

Airborne LiDAR Point Cloud Processing for Archaeology. Pipeline and QGIS Toolbox

2021

View full text Add to dashboard Cite

The use of topographic airborne LiDAR data has become an essential part of archaeological prospection. However, as a step towards theoretically aware, impactful, and reproducible research, a more rigorous and transparent method of data processing is required. To this end, we set out to create a processing pipeline for archaeology-specific point cloud processing and derivation of products that are optimized for general-purpose data. The proposed pipeline improves on ground and building point cloud classification. The main area of innovation in the proposed pipeline is raster grid interpolation. We have improved the state-of-the-art by introducing a hybrid interpolation technique that combines inverse distance weighting with a triangulated irregular network with linear interpolation. State-of-the-art solutions for enhanced visualizations are included and essential metadata and paradata are also generated. In addition, we have introduced a QGIS plug-in that implements the pipeline as a one-step process. It reduces the manual workload by 75 to 90 percent and requires no special skills other than a general familiarity with the QGIS environment. It is intended that the pipeline and tool will contribute to the white-boxing of archaeology-specific airborne LiDAR data processing. In discussion, the role of data processing in the knowledge production process is explored.

show abstract

“…Ref. [31] proposed an interpretable spatial-spectral reconstruction network, consisting of cross-mode message inserting, spatial reconstruction network, and spectral reconstruction network, to achieve the efficient fusion of hyperspectral and multispectral image. With respect to hyperspectral snapshot compressive reconstruction, Ref.…”

Section: Related Workmentioning

confidence: 99%

Hyperspectral Snapshot Compressive Imaging with Non-Local Spatial-Spectral Residual Network

et al. 2021

View full text Add to dashboard Cite

Snapshot Compressive Imaging is an emerging technology that is based on compressive sensing theory to achieve high-efficiency hyperspectral data acquisition. The core problem of this technology is how to reconstruct 3D hyperspectral data from the 2D snapshot measurement in a fast and high-quality manner. In this paper, we propose a novel deep network, which consists of the symmetric residual module and the non-local spatial-spectral attention module, to learn the reconstruction mapping in a data-driven way. The symmetric residual module uses symmetric residual connections to improve the potential of interaction between convolution operations and further promotes the fusion of local features. The non-local spatial-spectral attention module is designed to capture the non-local spatial-spectral correlation in the hyperspectral image. Specifically, this module calculates the channel attention matrix to capture the global correlations between all of the spectral channels, and it fuses the channel attention attained feature maps and the spatial attention weighted features as the module output, thus both of the spatial-spectral correlations of hyperspectral images can be fully utilized for reconstruction. In addition, a compound loss, including the reconstruction loss, the measurement loss, and the cosine loss, is designed to guide the end-to-end network learning. We experimentally evaluate the proposed method on simulation and real datasets. The experimental results show that the proposed network outperforms the competing methods in terms of the reconstruction quality and running time.

show abstract

SSR-NET: Spatial–Spectral Reconstruction Network for Hyperspectral and Multispectral Image Fusion

Cited by 107 publications

References 37 publications

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Airborne LiDAR Point Cloud Processing for Archaeology. Pipeline and QGIS Toolbox

Hyperspectral Snapshot Compressive Imaging with Non-Local Spatial-Spectral Residual Network

Contact Info

Product

Resources

About