Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation

Zhuang, Zhuangwei; Li, Rong; Li, Yuanqing; Jia, Kui; Wang, Qicheng; Tan, Mingkui

doi:10.1109/iccv48922.2021.01597

Cited by 113 publications

(54 citation statements)

References 54 publications

(99 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Polarimetric optical cues [11], [82], [83] and event-driven priors [12], [84], [85] are often intertwined for robust perception under adverse conditions. In automated driving, optical flow [86] and LiDAR data [87] are also incorporated for enhanced semantic road scene understanding. However, most of these works only address a single sensing-modality combination scenario.…”

Section: Multi-modal Semantic Segmentationmentioning

confidence: 99%

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Liu¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The performance of semantic segmentation of RGB images can be advanced by exploiting informative features from supplementary modalities. In this work, we propose CMX, a vision-transformer-based cross-modal fusion framework for RGB-X semantic segmentation. To generalize to different sensing modalities encompassing various uncertainties, we consider that comprehensive crossmodal interactions should be provided. CMX is built with two streams to extract features from RGB images and the complementary modality (X-modality). In each feature extraction stage, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate the feature of the current modality by combining the feature from the other modality, in spatial-and channel-wise dimensions. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix them for the final semantic prediction. FFM is constructed with a cross-attention mechanism, which enables exchange of long-range contexts, enhancing both modalities' features at a global level. Extensive experiments show that CMX generalizes to diverse multi-modal combinations, achieving state-of-the-art performances on four RGB-Depth benchmarks, as well as RGB-Thermal and RGB-Polarization datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish a RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. Code is available at https://github.com/huaaaliu/RGBX Semantic Segmentation

show abstract

Section: Multi-modal Semantic Segmentationmentioning

confidence: 99%

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Liu¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…monocular RGB images and point clouds. This is also inspired by multi-sensor-based methods on object tracking [55] , object detection [30,36] and point cloud segmentation [22,56]. In addition, we detect noisy labels and propose a noisy-label-aware training scheme to reduce the negative effect of noisy labels on the training, different from existing methods.…”

Section: Related Workmentioning

confidence: 99%

Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels

Li¹,

Cheng²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a novel scene flow method that captures 3D motions from point clouds without relying on ground-truth scene flow annotations. Due to the irregularity and sparsity of point clouds, it is expensive and time-consuming to acquire ground-truth scene flow annotations. Some state-of-the-art approaches train scene flow networks in a self-supervised learning manner via approximating pseudo scene flow labels from point clouds. However, these methods fail to achieve the performance level of fully supervised methods, due to the limitations of point cloud such as sparsity and lacking color information. To provide an alternative, we propose a novel approach that utilizes monocular RGB images and point clouds to generate pseudo scene flow labels for training scene flow networks. Our pseudo label generation module infers pseudo scene labels for point clouds by jointly leveraging rich appearance information in monocular images and geometric information of point clouds. To further reduce the negative effect of noisy pseudo labels on the training, we propose a noisy-labelaware training scheme by exploiting the geometric relations of points. Experiment results show that our method not only outperforms state-of-the-art self-supervised approaches but also outperforms some supervised approaches that use accurate ground-truth flows.

show abstract

“…Caltagirone et al [ 25 ] developed a novel multimodal system for road detection by fusing LiDAR and camera data, and the cross function convolutional neural network (FCN) architecture was introduced. Zhuang et al [ 26 ] investigated a collaborative fusion scheme called perception-aware multi-sensor fusion to exploit perceptual information from two modalities that included appearance information from RGB images and the spatio-depth information from point clouds, which achieved the 3D semantic segmentation. An offline LiDAR and camera fusion method was proposed by Zhen et al [ 27 ] to build dense and accurate 3D models.…”

Section: Related Workmentioning

confidence: 99%

Unifying Obstacle Detection, Recognition, and Fusion Based on the Polarization Color Stereo Camera and LiDAR for the ADAS

Long

Han²,

Wang

et al. 2022

Sensors

View full text Add to dashboard Cite

The perception module plays an important role in vehicles equipped with advanced driver-assistance systems (ADAS). This paper presents a multi-sensor data fusion system based on the polarization color stereo camera and the forward-looking light detection and ranging (LiDAR), which achieves the multiple target detection, recognition, and data fusion. The You Only Look Once v4 (YOLOv4) network is utilized to achieve object detection and recognition on the color images. The depth images are obtained from the rectified left and right images based on the principle of the epipolar constraints, then the obstacles are detected from the depth images using the MeanShift algorithm. The pixel-level polarization images are extracted from the raw polarization-grey images, then the water hazards are detected successfully. The PointPillars network is employed to detect the objects from the point cloud. The calibration and synchronization between the sensors are accomplished. The experiment results show that the data fusion enriches the detection results, provides high-dimensional perceptual information and extends the effective detection range. Meanwhile, the detection results are stable under diverse range and illumination conditions.

show abstract

Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation

Cited by 113 publications

References 54 publications

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels

Unifying Obstacle Detection, Recognition, and Fusion Based on the Polarization Color Stereo Camera and LiDAR for the ADAS

Contact Info

Product

Resources

About