CL3D: Camera-LiDAR 3D Object Detection With Point Feature Enhancement and Point-Guided Fusion

Abstract

Camera-LiDAR 3D object detection has been extensively investigated due to its significance for many real-world applications. However, there are still of great challenges to address the intrinsic data difference and perform accurate feature fusion among two modalities. To these ends, we propose a two-stream architecture termed as CL3D, that integrates with point enhancement module, point-guided fusion module with IoU-aware head for cross-modal 3D object detection. Specifically, pseudo LiDAR is firstly generated from RGB image, and point enhancement module (PEM) is then designed to enhance the raw LiDAR with pseudo point. Moreover, point-guided fusion module (PFM) is developed to find image-point correspondence at different resolutions, and incorporate semantic with geometric features in a point-wise manner. We also investigate the inconsistency between localization confidence and classification score in 3D detection, and introduce IoU-aware prediction head (IoU Head) for accurate box regression. Comprehensive experiments are conducted on publicly available KITTI dataset, and CL3D reports the outstanding detection performance compared to both single-and multi-modal 3D detectors, demonstrating its effectiveness and competitiveness.

3D Harmonic Loss: Towards Task-consistent and Time-friendly 3D Object Detection on Edge for Intelligent Transportation System

“…The inconsistency problem in 3D pointcloud object detection mechanisms causes low object detection reliability and quality of experience. For instance, several works [18], [19] attempted to solve it. However, their solutions, similar to PAA [5], consume extra time-cost for a supplementary branch of predicting IoU and additional operations in post-processing, which are not applicable for real-time environments.…”

“…Especially for real-time applications, as in V2X, there is a scope to design a cost-effective, taskfriendly and task-consistent detection solution with fast runtime and low error rate. On the one side, several researchers [18], [19] noticed the problem of task inconsistency. Nevertheless, their solutions relied on additional modules with extra inference time-cost, contrary to our application target of pursuing less computational burden.…”

“…Inspired by the solution in image domain [4], we extend the 2D solution to 3D detection and propose 3D harmonic loss, a task-consistent learning strategy for optimizing pointcloud-based 3D detectors. It is worth mentioning that our solution, 3D harmonic loss, not like previous solutions [18], [19], only works for model training and does not bring any extra time-cost to model inference. Secondly, a thorough mathematical analysis is conducted to explain and demonstrate the effectiveness of 3D harmonic loss.…”

V2VFormer: Vehicle-to-Vehicle Cooperative Perception With Spatial-Channel Transformer

“…As an essential ingredient, environmental perception provides an adequate comprehension of surroundings and participants from heterogeneous sensory data, thus gaining widespread popularity. The emergence in deep learning pushes a remarkable step for self-driving perception, and its accuracy/robustness has been significantly improved in several tasks such as object detection [2] [3] [4] [5], multi-object tracking [6], segmentation [7], etc. Despite its immense potential, single-agent perception suffers from occlusion, blind spot and sparse measurement (i.e., LiDAR point) challenges, and individual perspective easily causes an unreliable and uncertainty prediction especially in presence of Fig.…”

3D Harmonic Loss: Towards Task-Consistent and Time-Friendly 3D Object Detection on Edge for V2X Orchestration

“…tion solution with fast run-time and low error rate is required. Some researchers [19], [20] have noticed the problem of task inconsistency, but their solutions rely on additional modules that increase inference time, which contradicts our goal of reducing computational burden.…”