Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review

Yao, Shanliang; Guan, Runwei; Huang, Xiaoyu; Li, Zhuoxiao; Sha, Xiangyu; Yue, Yong; Lim, Eng Gee; Seo, Hyungjoon; Man, Ka Lok; Zhu, Xiaohui; Yue, Yutao

doi:10.1109/tiv.2023.3307157

Cited by 23 publications

(4 citation statements)

References 276 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of AD, sensor fusion improves the robustness and reduces the uncertainty of perception systems. In this Section, we define how sensor fusion can be performed by answering four main questions: what to fuse , where to fuse , when to fuse and how to fuse [ 5 ]. Then, a set of selected methods based on camera and automotive RADAR is revised.…”

Section: Related Workmentioning

confidence: 99%

Point Cloud Painting for 3D Object Detection with Camera and Automotive 3+1D RADAR Fusion

Montiel-Marín,

Llamazares,

Antunes

et al. 2024

Sensors

View full text Add to dashboard Cite

RADARs and cameras have been present in automotives since the advent of ADAS, as they possess complementary strengths and weaknesses but have been underlooked in the context of learning-based methods. In this work, we propose a method to perform object detection in autonomous driving based on a geometrical and sequential sensor fusion of 3+1D RADAR and semantics extracted from camera data through point cloud painting from the perspective view. To achieve this objective, we adapt PointPainting from the LiDAR and camera domains to the sensors mentioned above. We first apply YOLOv8-seg to obtain instance segmentation masks and project their results to the point cloud. As a refinement stage, we design a set of heuristic rules to minimize the propagation of errors from the segmentation to the detection stage. Our pipeline concludes by applying PointPillars as an object detection network to the painted RADAR point cloud. We validate our approach in the novel View of Delft dataset, which includes 3+1D RADAR data sequences in urban environments. Experimental results show that this fusion is also suitable for RADAR and cameras as we obtain a significant improvement over the RADAR-only baseline, increasing mAP from 41.18 to 52.67 (+27.9%).

show abstract

Section: Related Workmentioning

confidence: 99%

Point Cloud Painting for 3D Object Detection with Camera and Automotive 3+1D RADAR Fusion

Montiel-Marín,

Llamazares,

Antunes

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…Geng et al [45] discuss advancements in deep learning applied to radar perception, although their focus does not specifically center on odometry or place recognition. Conversely, the review conducted by Yao et al [4] revolves around radar-vision fusion specifically for object detection and segmentation.…”

Section: Related Workmentioning

confidence: 99%

“…Despite the considerable research on VSLAM techniques, they encounter difficulties in adverse weather conditions like heavy rain or fog, and in varying lighting situations such as intense sunlight or during the night. These challenges extend to issues concerning accurate depth perception [4,5].…”

Section: Introductionmentioning

confidence: 99%

Survey of Deep Learning-Based Methods for FMCW Radar Odometry and Ego-Localization

Brune,

Meisen,

Pomp

2024

Applied Sciences

View full text Add to dashboard Cite

This paper provides an in-depth review of deep learning techniques to address the challenges of odometry and global ego-localization using frequency modulated continuous wave (FMCW) radar sensors. In particular, we focus on the prediction of odometry, which involves the determination of the ego-motion of a system by external sensors, and loop closure detection, which concentrates on the determination of the ego-position typically on an existing map. We initially emphasize the significance of these tasks in the context of radar sensors and underscore the motivations behind them. The subsequent sections delve into the practical implementation of deep learning approaches, strategically designed to effectively address the aforementioned challenges. We primarily focus on spinning and automotive radar configurations within the domain of autonomous driving. Additionally, we introduce publicly available datasets that have been instrumental in addressing these challenges and analyze the importance and struggles of current methods used for radar based odometry and localization. In conclusion, this paper highlights the distinctions between the addressed tasks and other radar perception applications, while also discussing their differences from challenges posed by alternative sensor modalities. The findings contribute to the ongoing discourse on advancing radar sensor capabilities through the application of deep learning methodologies, particularly in the context of enhancing odometry and ego-localization for autonomous driving applications.

show abstract

“…Recent advancements in this field h ave i ntroduced novel approaches to fuse camera and radar data [31]. For instance, Simple-BEV [23] lifts image pixels to 3D voxels and concatenates them with radar BEV features before reducing the height dimension.…”

Section: B 3d Object Detection With Camera and Ordinary Automotive Ra...mentioning

confidence: 99%

LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion

Xiong,

Liu,

Huang

et al. 2024

2024 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review

Cited by 23 publications

References 276 publications

Point Cloud Painting for 3D Object Detection with Camera and Automotive 3+1D RADAR Fusion

Point Cloud Painting for 3D Object Detection with Camera and Automotive 3+1D RADAR Fusion

Survey of Deep Learning-Based Methods for FMCW Radar Odometry and Ego-Localization

LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion

Contact Info

Product

Resources

About