Real-Time Vehicle Detection Framework Based on the Fusion of LiDAR and Camera

Abstract

Vehicle detection is essential for driverless systems. However, the current single sensor detection mode is no longer sufficient in complex and changing traffic environments. Therefore, this paper combines camera and light detection and ranging (LiDAR) to build a vehicle-detection framework that has the characteristics of multi adaptability, high real-time capacity, and robustness. First, a multi-adaptive high-precision depth-completion method was proposed to convert the 2D LiDAR sparse depth map into a dense depth map, so that the two sensors are aligned with each other at the data level. Then, the You Only Look Once Version 3 (YOLOv3) real-time object detection model was used to detect the color image and the dense depth map. Finally, a decision-level fusion method based on bounding box fusion and improved Dempster–Shafer (D–S) evidence theory was proposed to merge the two results of the previous step and obtain the final vehicle position and distance information, which not only improves the detection accuracy but also improves the robustness of the whole framework. We evaluated our method using the KITTI dataset and the Waymo Open Dataset, and the results show the effectiveness of the proposed depth completion method and multi-sensor fusion strategy.

Obstacle Detection Using a Facet-Based Representation from 3-D LiDAR Measurements

“…We compared the results of facet detection IoU from the adapted algorithm [ 20 ] with our proposed approach on the following classes: car, cyclist, misc, pedestrian, truck, and van. For the person sitting class, the clustering algorithm takes other objects into account, e.g., a table, and evaluation is not correct.…”

“…The authors of [ 20 ] propose a real-time framework for object detection that combines camera and LiDAR sensors. The point cloud from LiDAR is converted into a dense depth map, which is aligned to the camera image.…”

Frustum-PointPillars: A Multi-Stage Approach for 3D Object Detection using RGB Camera and LiDAR

“…The detections from the independent networks are then fused using statistical methods or a smaller network. The merit of late fusion approaches is that in case of the failure of one sensor, the detections from another sensor can still be used, bearing a reduction in the accuracy [13]. Nevertheless, sometimes it can be difficult to perform a certain task efficiently in a redundant way due to the limitations of the sensor modalities, for example, 3D object detection using a monocular camera.…”

An Internet of Vehicles (IoV) Access Gateway Design Considering the Efficiency of the In-Vehicle Ethernet Backbone

“…To obtain more data for improved object recognition, autonomous vehicles attempt to apply high resolution cameras or multi-channel LiDAR sensors. In addition, they attempt to combine several cameras or multi-channel LiDAR sensors [ 2 , 3 ]. Thus, massive amounts of data are generated and transmitted from the vehicle devices.…”

“…In particular, massive data is generated by cameras and light detection and ranging (LiDAR) sensors for autonomous driving and by multimedia for infotainment services. Therefore, a high-speed IVN backbone is needed for this kind of data transmission [ 1 , 2 , 3 ].…”