NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion

Abstract

The fusion of high-resolution multispectral (HrMSI) and low-resolution hyperspectral images (LrHSI) has been acknowledged as a promising method for generating a highresolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of the underlying materials. In order to improve the fusion of LrHSI and HrMSI performance, in this article, we propose a novel Nonnegative Matrix Factorization Inspired Deep Unrolling Networks, dubbed NMF-DuNet, for fusing LrHSI and HrMSI. For this aim, initially, a variational fusion model regularized by non-negative sparse prior is proposed and then is solved through the gradient descent optimization method and unrolled towards the deep network. The nonnegative coefficient matrices and orthogonal of the proposed transform coefficients constraints are both incorporated into the proposed method. Moreover, the fusion of HrMSI and LrHSI heavily depends on an imaging model that explains the degeneracy of HSI in the spectral and spatial regions. Practically, the imaging model is often unknown. The degradation model is represented implicitly via a proposed network, and both the degradation model and sparse priors are jointly optimized through the training process of the proposed network. Instead of being hand-crafted, all the parameters of NMF-DuNet are learned end-to-end. Compared to the previous state-of-the-art model-based and learning-based fusion approaches, the hardware-friendly proposed NMF-DuNet outperforms both the model-based and learning-based fusion approaches and requires a far smaller number of trainable parameters and storage space while preserving real-time performance.

Utilizing Deep Convolutional Neural Networks and Non-Negative Matrix Factorization for Multi-Modal Image Fusion

“…This layer extracts more intricate and detailed information from the pooled feature map, capturing finer patterns and features. To minimize the training complexity and memory usage, a lightweight network structure is employed for this convolutional layer [22].…”

Model-Guided Deep Unfolded Fusion Network With Nonlocal Spatial-Spectral Priors for Hyperspectral Image Super-Resolution

“…In our proposed network, we simulate the multiplication process by convolutional filters as stated in [57]; thereby, we use a convolutional layer with filter size z × z and the number of filters K that can estimate D T automatically as a trainable operator through the learning process. Meanwhile, we can obtain the spatial degraded coefficients matrix by applying the degradation blur matrix B to the obtained A from (10), according to the imaging model of (2).…”

Enhanced Hyperspectral Sharpening through Improved Relative Spectral Response Characteristic (R-SRC) Estimation for Long-Range Surveillance Applications

“…However, fusion of high-spatialresolution multispectral images (HRMSI) or high-resolution panchromatic images with low-spatial-resolution hyperspectral images (LRHSI) has been a highly popular and more effective approach for the recovery of HRHSI, with accuracy far better than that using single LRHSI as the input [3,[5][6][7][8][9][10][14][15][16]. There are a number of fusion schemes designed to implement this kind of SR: the spectral unmixing convex programming approach using coupled non-negative factorization (CNMF) [17][18][19][20], various forms of coupled matrix and tensor factorization optimizations (CMTF) [21][22][23], nonlinear unmixing through intrinsic and extrinsic priors [24], and the implementation of CNMF in deep learning (DL) network architectures [5,[25][26][27], which have been reported in the literature over the past couple of decades.…”