ABMDRNet: Adaptive-weighted Bi-directional Modality Difference Reduction Network for RGB-T Semantic Segmentation

Zhang, Qiang; Zhao, Shenlu; Luo, Yongjiang; Zhang, Dingwen; Huang, Nianchang; Han, Jungong

doi:10.1109/cvpr46437.2021.00266

Cited by 90 publications

(65 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, it remains unclear whether potential improvements on RGB-X semantic segmentation can be materialized via vision transformers. Besides, while some previous works [7], [8] use a simple global multi-modal interaction strategy, it does not generalize well across different sensing data combinations [10]. We hypothesize that for RGB-X semantic segmentation with various noises and uncertainties, comprehensive cross-modal interactions should be provided, to fully exploit the potential of cross-modal complementary features.…”

Section: Introductionmentioning

confidence: 94%

“…For example, depth measurement can help identify the boundaries of objects and offer geometric information of dense scene elements [7], [8]. Thermal images facilitate to discern different objects through their specific infrared imaging [9], [10]. Besides, polarimetric-and event information are advantageous for perception in specularand dynamic real-world scenes [11], [12].…”

Section: Introductionmentioning

confidence: 99%

“…This is a typical strategy in RGB-D semantic segmentation with depth-guided operators [14], [15], [16], which are usually elaborately tailored for specific modalities (e.g., depth), yet hard to be extended to operate with other modality combinations. The second type of approaches [7], [8], [10], [17] deploys two backbones to perform feature extraction from RGB-and another modality separately, then fuses the extracted two features into one feature for semantic prediction (Fig. 2(b)).…”

Section: Introductionmentioning

confidence: 99%

“…This structure offers the extensibility to replace the other modalstream with alternative complementary modalities. However, most previous methods are predominantly crafted for specific modalities, e.g., they only focus on tackling RGB-D scene parsing [7], [8], [18] or RGB-T segmentation [10], [17], [19]. While inspiring, they fail to work well on different multi-modal scenarios.…”

Section: Introductionmentioning

confidence: 99%

“…While inspiring, they fail to work well on different multi-modal scenarios. For example, according to [10], [19], ACNet [7] and SA-Gate [8], designed for RGB-D data, perform less satisfactorily in RGB-T tasks. To enable robust real-world scene understanding, a general and flexible architecture suitable for a diverse mix of sensor data combinations enhancing RGB segmentation (Fig.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Liu¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The performance of semantic segmentation of RGB images can be advanced by exploiting informative features from supplementary modalities. In this work, we propose CMX, a vision-transformer-based cross-modal fusion framework for RGB-X semantic segmentation. To generalize to different sensing modalities encompassing various uncertainties, we consider that comprehensive crossmodal interactions should be provided. CMX is built with two streams to extract features from RGB images and the complementary modality (X-modality). In each feature extraction stage, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate the feature of the current modality by combining the feature from the other modality, in spatial-and channel-wise dimensions. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix them for the final semantic prediction. FFM is constructed with a cross-attention mechanism, which enables exchange of long-range contexts, enhancing both modalities' features at a global level. Extensive experiments show that CMX generalizes to diverse multi-modal combinations, achieving state-of-the-art performances on four RGB-Depth benchmarks, as well as RGB-Thermal and RGB-Polarization datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish a RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. Code is available at https://github.com/huaaaliu/RGBX Semantic Segmentation

show abstract

Section: Introductionmentioning

confidence: 94%