FusAtNet: Dual Attention based SpectroSpatial Multimodal Fusion Network for Hyperspectral and LiDAR Classification

Abstract

With recent advances in sensing, multimodal data is becoming easily available for various applications, especially in remote sensing (RS), where many data types like multispectral imagery (MSI), hyperspectral imagery (HSI), Li-DAR etc. are available. Effective fusion of these multisource datasets is becoming important, for these multimodality features have been shown to generate highly accurate land-cover maps. However, fusion in the context of RS is non-trivial considering the redundancy involved in the data and the large domain differences among multiple modalities. In addition, the feature extraction modules for different modalities hardly interact among themselves, which further limits their semantic relatedness. As a remedy, we propose a feature fusion and extraction framework, namely FusAtNet, for collective land-cover classification of HSIs and LiDAR data in this paper. The proposed framework effectively utilizses HSI modality to generate an attention map using "self-attention" mechanism that highlights its own spectral features. Similarly, a "crossattention" approach is simultaneously used to harness the LiDAR derived attention map that accentuates the spatial features of HSI. These attentive spectral and spatial representations are then explored further along with the original data to obtain modality-specific feature embeddings. The modality oriented joint spectro-spatial information thus obtained, is subsequently utilized to carry out the land-cover classification task. Experimental evaluations on three HSI-LiDAR datasets show that the proposed method achieves the state-of-the-art classification performance, including on the largest HSI-LiDAR dataset available, University of Houston (Data Fusion Contest-2013), opening new avenues in multimodal feature fusion for classification. * Equal Contribution Corresponding Author Classification Module Fusion Black Box HSI (Visible shown) LiDAR Classification Map Fused Joint Representation Figure 1. Generic schematic of a multimodal fusion based classification task. The objective is to effectively combine the two modalities (hereby HSI and LiDAR) such that the resultant representation has rich, fused features that are relevant and robust enough for accurate classification.

Relative Total Variation Structure Analysis-Based Fusion Method for Hyperspectral and LiDAR Data Classification

“…(3) The feature-level fusion method proposed in this paper only contains a single spatial feature. Some researchers have found that the fusion of multiple features can improve the accuracy of land cover classification [63,64]. Therefore, the fusion of multiple features to improve the usability of remote sensing images will be considered in future work.…”

Critical insights into modern hyperspectral image applications through deep learning

“…Mohla et al (2020) focused on multimodal feature fusion for classification. The authors focused on the "self‐attention" and "cross‐attention”mechanism to explore spectral and spatial features from HSI and LiDAR.…”

EMFNet: Enhanced Multisource Fusion Network for Land Cover Classification

“…• SVM: It adopts the fused hyperspectral data and LiDAR data as input and SVM as the classifier. Since the work in [32] also did the same experiment, it is reasonable to cite the results from [32]. • CHOTF: It is a Coupled Higher-Order Tensor Factorization (CHOTF) model for hyperspectral and LiDAR fusion proposed in [33].…”