Deep Dual-Modal Traffic Objects Instance Segmentation Method Using Camera and LIDAR Data for Autonomous Driving

Abstract

Recent advancements in environmental perception for autonomous vehicles have been driven by deep learning-based approaches. However, effective traffic target detection in complex environments remains a challenging task. This paper presents a novel dual-modal instance segmentation deep neural network (DM-ISDNN) by merging camera and LIDAR data, which can be used to deal with the problem of target detection in complex environments efficiently based on multi-sensor data fusion. Due to the sparseness of the LIDAR point cloud data, we propose a weight assignment function that assigns different weight coefficients to different feature pyramid convolutional layers for the LIDAR sub-network. We compare and analyze the adaptations of early-, middle-, and late-stage fusion architectures in depth. By comprehensively considering the detection accuracy and detection speed, the middle-stage fusion architecture with a weight assignment mechanism, with the best performance, is selected. This work has great significance for exploring the best feature fusion scheme of a multi-modal neural network. In addition, we apply a mask distribution function to improve the quality of the predicted mask. A dual-modal traffic object instance segmentation dataset is established using a 7481 camera and LIDAR data pairs from the KITTI dataset, with 79,118 manually annotated instance masks. To the best of our knowledge, there is no existing instance annotation for the KITTI dataset with such quality and volume. A novel dual-modal dataset, composed of 14,652 camera and LIDAR data pairs, is collected using our own developed autonomous vehicle under different environmental conditions in real driving scenarios, for which a total of 62,579 instance masks are obtained using semi-automatic annotation method. This dataset can be used to validate the detection performance under complex environmental conditions of instance segmentation networks. Experimental results on the dual-modal KITTI Benchmark demonstrate that DM-ISDNN using middle-stage data fusion and the weight assignment mechanism has better detection performance than single- and dual-modal networks with other data fusion strategies, which validates the robustness and effectiveness of the proposed method. Meanwhile, compared to the state-of-the-art instance segmentation networks, our method shows much better detection performance, in terms of AP and F1 score, on the dual-modal dataset collected under complex environmental conditions, which further validates the superiority of our method.

ETLi: Efficiently annotated traffic LiDAR dataset using incremental and suggestive annotation

“…However, the lack of variety in terms of weather and light conditions limits its potential. Therefore, the Dual-Modal Dataset [22], which includes paired LiDAR and RGB image data from the Kitti dataset, was released for traffic-object instance segmentation.…”

WARA-PS: a research arena for public safety demonstrations and autonomous collaborative rescue robotics experimentation

“…However, also for (semi-)autonomous systems, visual information can be considered for GPS-independent navigation and positioning purposes. While there are quite many (annotated) data sets and annotation tools suitable for, among others, machine learning approaches in the area of autonomous vehicles on land ([ 63 , 64 ], for example), there are relatively few starting points to work with image classification approaches in maritime environments. To mitigate that, we investigated the possibility to create a semi-automated tool for image annotation based on image sets gathered in the previously described trial and data collection runs in WARA-PS [ 65 ].…”

A New Density-Based Clustering Method Considering Spatial Distribution of Lidar Point Cloud for Object Detection of Autonomous Driving

“…In this condition, a large clustering radi preferred to avoid over-segmentation. To show this problem more intuitively, the traditional DSC was tested on the K dataset [34], with the clustering radius and the minimum number of points (denote To show this problem more intuitively, the traditional DSC was tested on the KITTI dataset [34], with the clustering radius and the minimum number of points (denoted by MinPts) set to 0.15 m and 5, respectively. Some typical wrong segmentations are shown in Figure 4.…”

“…Specifically, the inherent attributes of the point cloud, h and v, are calculated by the position of the sample point; the mesh width in the grid map, w, has been given in the pre-processing stage; the upper boundary of the semi-major, βL, depends on the maximum length of the detected object according to the design concept from Section 3.2; and the linear coefficients α and β need to be studied. Thus, the theoretical relation between MinPts, α and β is studied, and numerical optimization of parameters is analyzed in the KITTI dataset [34] by considering the comprehensive performance.…”