Vehicle Detection Based on Information Fusion of mmWave Radar and Monocular Vision

Abstract

Single sensors often fail to meet the needs of practical applications due to their lack of robustness and poor detection accuracy in harsh weather and complex environments. A vehicle detection method based on the fusion of millimeter wave (mmWave) radar and monocular vision was proposed to solve this problem in this paper. The method successfully combines the benefits of mmWave radar for measuring distance and speed with the vision for classifying objects. Firstly, the raw point cloud data of mmWave radar can be processed by the proposed data pre-processing algorithm to obtain 3D detection points with higher confidence. Next, the density-based spatial clustering of applications with noise (DBSCAN) clustering fusion algorithm and the nearest neighbor algorithm were also used to correlate the same frame data and adjacent frame data, respectively. Then, the effective targets from mmWave radar and vision were matched under temporal-spatio alignment. In addition, the successfully matched targets were output by using the Kalman weighted fusion algorithm. Targets that were not successfully matched were marked as new targets for tracking and handled in a valid cycle. Finally, experiments demonstrated that the proposed method can improve target localization and detection accuracy, reduce missed detection occurrences, and efficiently fuse the data from the two sensors.

A Robust Target Detection Algorithm Based on the Fusion of Frequency-Modulated Continuous Wave Radar and a Monocular Camera

“…As a limitation of the software, the average inference time per frame of both algorithms is much larger than the actual acquisition time, which makes it difficult to achieve real-time processing. However, the inference time of the proposed algorithm increased by only 0.04 s compared to [41]. Therefore, combining Tables 3 and 4, we can conclude that the proposed algorithm is able to achieve superior target detection performance at the expense of a certain time and space.…”

“…Moreover, the F1 score is almost unchanged under different lighting environments, which suggests that the performance of our algorithm is consistent under various environments. In addition, the proposed fusion algorithm outperforms [41] in all aspects. This is attributed to the fact that we used the lane detection results to assist the fusion of the radar and camera, whereas [41] had difficulty in solving the problem of mutual coverage of the bounding boxes of different targets on adjacent lanes.…”

“…In addition, the proposed fusion algorithm outperforms [41] in all aspects. This is attributed to the fact that we used the lane detection results to assist the fusion of the radar and camera, whereas [41] had difficulty in solving the problem of mutual coverage of the bounding boxes of different targets on adjacent lanes. Finally, in order to analyze the algorithm's inference time and running memory, we processed 199 frames of data from the scene shown in Figure 23.…”

“…In order to make consistent judgments on the targets detected by the radar and camera, it is necessary to generate the corresponding vehicle ROI based on each pixel point mapped by the radar in the image [41]. The vehicle's external dimensions specify that the aspect ratio of the vehicle's rear should range from 0.7 to 1.3.…”

“…In the same experimental scenario, comparison experiments were performed between the results of camera-only detection, the proposed fusion algorithm, and another decision-level fusion method [41]. As the methods of cameraonly detection and [41] need to pre-set the lane width to filter out the interfering targets, we set their lateral effective range to [−6.0 m, 1.3 m]. On the contrary, the proposed method does not require us to set the lane threshold.…”