Deep learning in multimodal remote sensing data fusion: A comprehensive review

Li, Jiaxin; Hong, Danfeng; Gao, Lianru; Yao, Jing; Zheng, Ke; Zhang, Bing; Chanussot, Jocelyn

doi:10.1016/j.jag.2022.102926

Cited by 144 publications

(69 citation statements)

References 197 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, the integration of machine and deep learning algorithms specifically designed for remote sensing applications, such as convolutional neural networks and vision transformers [34,35], can help enhance the performance and capabilities of visual models. These methods can improve the VLM's ability to recognize and analyze complex patterns, structures, and features in remote sensing images, leading to more accurate and reliable results.…”

Section: Discussion: Improving Visual Language Models For Remote Sens...mentioning

confidence: 99%

The Potential of Visual ChatGPT For Remote Sensing

Osco¹,

Lopes²,

Gonçalves³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advancements in Natural Language Processing (NLP), particularly in Large Language Models (LLMs), associated with deep learning-based computer vision techniques, have shown substantial potential for automating a variety of tasks. One notable model is Visual ChatGPT, which combines ChatGPT’s LLM capabilities with visual computation to enable effective image analysis. The model’s ability to process images based on textual inputs can revolutionize diverse fields. However, its application in the remote sensing domain remains unexplored. This is the first paper to examine the potential of Visual ChatGPT, a cutting-edge LLM founded on the GPT architecture, to tackle the aspects of image processing related to the remote sensing domain. Among its current capabilities, Visual ChatGPT can generate textual descriptions of images, perform canny edge and straight line detection, and conduct image segmentation. These offer valuable insights into image content and facilitate the interpretation and extraction of information. By exploring the applicability of these techniques within publicly available datasets of satellite images, we demonstrate the current model’s limitations in dealing with remote sensing images, highlighting its challenges and future prospects. Although still in early development, we believe that the combination of LLMs and visual models holds a significant potential to transform remote sensing image processing, creating accessible and practical application opportunities in the field.

show abstract

Section: Discussion: Improving Visual Language Models For Remote Sens...mentioning

confidence: 99%

The Potential of Visual ChatGPT For Remote Sensing

Osco¹,

Lopes²,

Gonçalves³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The pixel-wise classification of HSI, which appears as an important issue of HSI processing technology, achieves a phenomenal interest of researchers and has been studied by many scholars in recent years [9,10]. The purpose of the pixelwise classification is to assign a unique category label to each pixel of the HSI dataset.…”

Section: Introductionmentioning

confidence: 99%

Pyramidal Multiscale Convolutional Network With Polarized Self-Attention for Pixel-Wise Hyperspectral Image Classification

Wang

Liu

et al. 2023

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

In recent years, pixel-wise hyperspectral image (HSI) classification has received growing attention in the field of remote sensing. Plenty of spectral-spatial convolutional neural network (CNN) methods with diverse attention mechanisms have been proposed for HSI classification due to the attention mechanisms being able to provide more flexibility over standard convolutional blocks. However, it remains a challenge to effectively extract multi-scale features of high-resolution HSI in a real-world complex environment. In this paper, we propose a pyramidal multi-scale spectral-spatial convolutional network with polarized self-attention for pixel-wise HSI classification. It contains three stages: channel-wise feature extraction network, spatial-wise feature extraction network, and classification network, which are used to extract spectral features, extract spatial features, and generate classification results, respectively. Pyramidal convolutional blocks and polarized attention blocks are combined to extract spectral and spatial features of HSI. Furthermore, residual aggregation and one-shot aggregation are employed to better converge the network. Experimental results on several public HSI datasets demonstrate that the proposed network outperforms other related methods.

show abstract

“…There has been a trend towards incorporating multiple modalities in aerial object classification to improve performance and robustness. [11][12][13][14] This includes different types of imagery, such as RGB, infrared, hyperspectral, multispectral, synthetic aperture radar (SAR), and light detection and ranging, as well as other data sources such as terrain maps and building footprints. Multimodal approaches have shown promising results, as they leverage complementary information from different sources to provide a more complete and accurate representation of the scene.…”

Section: Introductionmentioning

confidence: 99%

“…Multimodal approaches have shown promising results, as they leverage complementary information from different sources to provide a more complete and accurate representation of the scene. 11,15 However, collecting data from multiple modalities can be challenging and expensive due to the requirement of specialized equipment, atmospheric conditions, limitation of individual modalities to probe a scene, data integration from modalities with different spatial and spectral resolutions, and annotation challenges for obtaining ground truth. 16,17 Researchers often encounter the issue of limited paired multimodal data for training an end-to-end multimodal fusion network, 16 where paired multimodal data samples simultaneously agree with the following conditions: 1) spatial and temporal correspondence and 2) synchronous occurrence/availability.…”

Section: Introductionmentioning

confidence: 99%

“…Based upon these underlying conditions, the following research questions arise: 1) is it feasible and computationally advantageous to fuse legacy unimodal networks pre-trained on unpaired multimodal data samples and 2) what are the optimal approaches to train a fusion network if all paired multimodal data is available? Example approaches for training fusion networks are: i) training a multimodal network from scratch on all the paired multimodal data 11,[18][19][20] and ii) training unimodal networks in phase one and using the corresponding feature extractors along with a fraction of paired multimodal data to fuse them in phase two. This leads to needing to understand what training paradigms perform the best (e.g., freezing or not freezing the feature extractor weights, data split between phase one and phase) in terms of accuracy and robustness against different noise profiles.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation