Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

Li, Jiaxin; Hong, Danfeng; Gao, Lianru; Yao, Jianquan; Zheng, Kan; Zhang, Bing; Chanussot, Jocelyn

doi:10.48550/arxiv.2205.01380

Cited by 7 publications

(6 citation statements)

References 137 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• The current implementation cannot take multi-modal inputs that belong to one task to generate a solution (Multi-modal Machine Learning). Humans can take images, data, sound, and smell together to complete the task, but it is still a challenge for the machine to merge different types of inputs together to solve the problem [44][45][46]. • When we started writing this paper, the Google research papers were published introducing the mutation of Neural Network (NN) to deal with multiple image classification tasks [15,16].…”

Section: Discussionmentioning

confidence: 99%

AIMS: An Automatic Semantic Machine Learning Microservice Framework to Support Biomedical and Bioengineering Research

O'Neill¹,

Kermanizadeh²

2023

Preprint

View full text Add to dashboard Cite

The fusion of machine learning, biomedical, and bioengineering research offers novel ways to understand, diagnose, and treat various health conditions. However, the complexities of biomedical and bioengineering data, coupled with the intricate process of developing and deploying machine learning solutions, often pose significant challenges to researchers in these fields. We introduce a new Automatic Semantic Machine Learning Microservice Framework (AIMS) that addresses these challenges by automating key aspects of the machine learning pipeline, from data preprocessing to model evaluation. Our proposed framework emphasizes model interpretability, the integration of domain knowledge, and the efficient handling of biomedical data. Furthermore, it incorporates self-supervised knowledge learning strategies, enabling the automatic generation, optimization, and continuous adaptation of machine learning models in response to new tasks and data. We illustrate the potential of our framework through three case studies in biomedical research, showcasing its capacity to streamline research processes, improve scientific exploration quality, and provide a foundation for significant breakthroughs.

show abstract

Section: Discussionmentioning

confidence: 99%

AIMS: An Automatic Semantic Machine Learning Microservice Framework to Support Biomedical and Bioengineering Research

O'Neill¹,

Kermanizadeh²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…R EMOTE sensing image classification task plays an essential role in earth observation, which could be used for analyzing critical information related to urban planning, natural resources management, climate change, environmental monitoring, and so on. Remote sensing data acquired from various sensors could exploit multiple physical characteristics of ground objects [1]- [5]. With the blooming development of remote sensing sensors, more and more researchers in the remote sensing community are active in the algorithm innovations to better extract the most valuable features among the multimodal remote sensing data [6].…”

Section: Introductionmentioning

confidence: 99%

Multimodal Attention-Aware Convolutional Neural Networks for Classification of Hyperspectral and LiDAR Data

Zhang

Gao

et al. 2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

Self Cite

View full text Add to dashboard Cite

The attention mechanism is one of the most influential ideas in the deep learning community, which has shown excellent efficiency in various computer vision tasks. Thus, this paper proposes the convolution neural network method with the attention mechanism to enhance the feature extraction of light detection and ranging (LiDAR) data. Meanwhile, our elaborately designed cascaded block contains a short path architecture beneficial for multi-stage information exchange. With the full exploitation of elevation information from LiDAR data and efficient utilization of the spatial-spectral information underlying hyperspectral data, our method provides a novel solution for multimodal feature fusion. Experiments are conducted on the Lidar and hyperspectral dataset provided by the 2013 IEEE GRSS Data Fusion Contest and multisource Trento dataset to demonstrate the effectiveness of the proposed method. The experimental results have shown the superior results of the proposed method on both LiDAR and multi-modality remote sensing data in comparison with several popular baselines.

show abstract

“…Currently, with the rise of space exploration technology, HSI is growing towards big data, and thus the proliferation of HSI requires accurate classification of more new scenes, which demands sufficient and accurate labelled samples [12,14,15]. However, it is extremely difficult to obtain thousands of labelled samples of HSI, which requires great human, material, and financial resources.…”

Section: Introductionmentioning

confidence: 99%

Deep transformer and few‐shot learning for hyperspectral image classification

Ran

Zhou

Hong

et al. 2023

CAAI Trans on Intel Tech

View full text Add to dashboard Cite

Recently, deep learning has achieved considerable results in the hyperspectral image (HSI) classification. However, most available deep networks require ample and authentic samples to better train the models, which is expensive and inefficient in practical tasks. Existing few‐shot learning (FSL) methods generally ignore the potential relationships between non‐local spatial samples that would better represent the underlying features of HSI. To solve the above issues, a novel deep transformer and few‐shot learning (DT‐FSL) classification framework is proposed, attempting to realize fine‐grained classification of HSI with only a few‐shot instances. Specifically, the spatial attention and spectral query modules are introduced to overcome the constraint of the convolution kernel and consider the information between long‐distance location (non‐local) samples to reduce the uncertainty of classes. Next, the network is trained with episodes and task‐based learning strategies to learn a metric space, which can continuously enhance its modelling capability. Furthermore, the developed approach combines the advantages of domain adaptation to reduce the variation in inter‐domain distribution and realize distribution alignment. On three publicly available HSI data, extensive experiments have indicated that the proposed DT‐FSL yields better results concerning state‐of‐the‐art algorithms.

show abstract

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

Cited by 7 publications

References 137 publications

AIMS: An Automatic Semantic Machine Learning Microservice Framework to Support Biomedical and Bioengineering Research

AIMS: An Automatic Semantic Machine Learning Microservice Framework to Support Biomedical and Bioengineering Research

Multimodal Attention-Aware Convolutional Neural Networks for Classification of Hyperspectral and LiDAR Data

Deep transformer and few‐shot learning for hyperspectral image classification

Contact Info

Product

Resources

About