Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving

Abstract

3D object detection is an important task in autonomous driving to perceive the surroundings. Despite the excellent performance, the existing 3D detectors lack the robustness to real-world corruptions caused by adverse weathers, sensor noises, etc., provoking concerns about the safety and reliability of autonomous driving systems. To comprehensively and rigorously benchmark the corruption robustness of 3D detectors, in this paper we design 27 types of common corruptions for both LiDAR and camera inputs considering real-world driving scenarios. By synthesizing these corruptions on public datasets, we establish three corruption robustness benchmarks-KITTI-C, nuScenes-C, and Waymo-C. Then, we conduct large-scale experiments on 24 diverse 3D object detection models to evaluate their corruption robustness. Based on the evaluation results, we draw several important findings, including: 1) motion-level corruptions are the most threatening ones that lead to significant performance drop of all models; 2) LiDAR-camera fusion models demonstrate better robustness; 3) camera-only models are extremely vulnerable to image corruptions, showing the indispensability of LiDAR point clouds. We release the benchmarks and codes at https://github.com/kkkcx/ 3D_Corruptions_AD. We hope that our benchmarks and findings can provide insights for future research on developing robust 3D object detection models.

The Effect of Camera Data Degradation Factors on Panoptic Segmentation for Automated Driving

“…Sensor Noise Model The camera sensors of AVs can suffer from many types of noises when they are impacted by internal or external variables [12]. Following the recent paper [12], we consider 3 types of noise: Gaussian noise, Uniform noise, and Impulse noise.…”

“…As these datasets are captured under good weather conditions with high visual quality, the data-driven panoptic segmentation models trained on them will show performance degradation when faced with various unseen or less-seen noise factors as are common in real-world scenarios. Examples of these degradation factors have been studied in details in [8], and include adverse weather [9], [10], [11], sensor internal noise [12], compression [13], [4] and unfavourable lighting scenarios [14], [15]. The lack of robustness in the existing panoptic segmentation methods can lead to safety problems in AAD functions.…”

“…The lack of robustness in the existing panoptic segmentation methods can lead to safety problems in AAD functions. Therefore, researchers have conducted experiments to compare the effect of various camera data degradation factors by using the evaluation with image classification [16], [4], object detection [12], [17], and semantic segmentation [18]. Nevertheless, none of the above tasks can provide the same level of detail as panoptic segmentation.…”

Benchmarking Adversarial Patch Against Aerial Detection

“…To comprehensively assess the robustness of these models, we conducted rigorous experiments involving a diverse set of classifiers and detectors, representing a wide range of mainstream methods. Through this extensive evaluation, we have uncovered insightful and intriguing findings that illuminate the relationship between the crafting of [32] IEEE Access ✓ ✕ ✕ ✕ ✕ ✕ 2018 [34] Computer Science Review ✓ ✕ ✕ ✕ ✕ ✕ 2018 [31] arXiv ✓ ✕ ✕ ✕ ✕ ✕ 2019 [33] Applied Science ✓ ✕ ✕ ✕ ✕ ✕ 2020 [42] ACM Computing Surveys ✓ ✕ ✕ ✕ ✕ ✕ 2021 [35] IEEE Access ✓ ✕ ✕ ✕ ✕ ✕ 2021 [41] ACM Computing Surveys ✓ ✕ ✕ ✕ ✕ ✕ 2021 [40] TII ✓ ✕ ✕ ✕ ✕ ✕ 2022 [47] arXiv ✓ * ✕ ✕ ✕ ✕ 2022 [48] INJOIT ✓ ✕ ✕ ✕ ✕ ✕ 2022 [49] Artificial Intelligence Review ✓ ✕ ✓ ✕ ✓ ✕ 2022 [39] TPAMI ✓ ✕ ✕ ✕ ✕ ✕ 2022 [38] TII ✕ ✕ ✕ ✕ ✕ ✕ 2022 [49] arXiv ✓ * ✕ ✕ ✕ ✕ 2022 [25] arXiv ✓ ✕ ✕ ✕ ✕ ✕ 2022 [44] arXiv ✓ * ✕ ✕ ✕ ✕ 2022 [45] arXiv * ✓ ✕ ✕ ✕ ✕ 2022 [37] Neurocomputing ✓ ✕ ✕ ✕ ✕ ✕ 2023 [50] ACM Computing Surveys * ✕ ✕ ✕ ✕ ✕ 2023 [28] arXiv ✓ ✕ ✕ ✕ ✕ ✕ 2023 [46] ICAI * ✓ ✕ ✕ ✕ ✕ Benchmarks 2020 [29] CVPR ✕ ✕ ✓ ✕ ✓ ✕ 2021 [27] arXiv ✕ ✕ ✓ ✕ ✓ ✓ 2022 [51] IJCAI ✕ ✕ ✕ ✓ ✓ ✕ 2022 [26] NIPS ✕ ✕ ✓ ✕ ✓ ✕ 2022 [52] arXiv ✕ ✕ ✕ ✓ ✓ ✕ 2022 [36] Pattern Recognition ✕ ✕ ✓ ✕ ✓ ✕ 2023 [30] arXiv ✕ ✕ ✓ ✕ ✓ ✓ 2023 [53] Pattern Recognition ✕ ✕ ✓ ✕ ✓ ✕ 2023 [54] CVPR…”