A*3D Dataset: Towards Autonomous Driving in Challenging Environments

Abstract

With the increasing global popularity of selfdriving cars, there is an immediate need for challenging realworld datasets for benchmarking and training various computer vision tasks such as 3D object detection. Existing datasets either represent simple scenarios or provide only day-time data. In this paper, we introduce a new challenging A*3D dataset which consists of RGB images and LiDAR data with significant diversity of scene, time, and weather. The dataset consists of high-density images (≈ 10 times more than the pioneering KITTI dataset), heavy occlusions, a large number of nighttime frames (≈ 3 times the nuScenes dataset), addressing the gaps in the existing datasets to push the boundaries of tasks in autonomous driving research to more challenging highly diverse environments. The dataset contains 39K frames, 7 classes, and 230K 3D object annotations. An extensive 3D object detection benchmark evaluation on the A*3D dataset for various attributes such as high density, day-time/night-time, gives interesting insights into the advantages and limitations of training and testing 3D object detection in real-world setting.

ROAD: The Road Event Awareness Dataset for Autonomous Driving

“…Among other notable multimodal datasets [18], [40] only consist of raw data without semantic labels, whereas [41] and [42] provide labels for location category and driving behaviour, respectively. The most recent multimodal large-scale AV datasets [43], [44], [45], [46], [47], [48] are significantly larger in terms of both data (also captured under varying weather conditions, e.g. by night or in the rain) and annotations (RGB, LiDAR/radar, 3D boxes).…”

“…Both 3D bounding boxes based on LiDAR data and 2D annotation on camera data for 4 objects classes are provided in Waymo [45]. In [46], using similar 3D annotation for 5 objects classes, the authors provide a more challenging dataset by adding more night-time scenarios using a faster-moving car. Amongst large-scale multimodal datasets, nuScenes [49], Lyft L5 [44], Waymo Open [45] and A*3D [46] are the most dominant ones in terms of number of instances, the use of high-quality sensors with different types of data (e.g., point clouds or 360 • RGB videos), and richness of the annotation providing both semantic information and 3D bounding boxes.…”

“…In [46], using similar 3D annotation for 5 objects classes, the authors provide a more challenging dataset by adding more night-time scenarios using a faster-moving car. Amongst large-scale multimodal datasets, nuScenes [49], Lyft L5 [44], Waymo Open [45] and A*3D [46] are the most dominant ones in terms of number of instances, the use of high-quality sensors with different types of data (e.g., point clouds or 360 • RGB videos), and richness of the annotation providing both semantic information and 3D bounding boxes. Furthermore, nuScenes [49], Argoverse [43] Lyft L5 [44] and KITTI-360 [48] provide contextual knowledge through human-annotated rich semantic maps, an important prior for scene understanding.…”

Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

“…In driving scenarios, a few methods have been proposed to detect [19] and vehicles' rear lights [20]. Contrary to these domainspecific methods, previous work also includes both methods designed for robustness to illumination changes, by employing domain-invariant representations [21], [22] or fusing information from complementary modalities and spectra [23], and datasets with adverse illumination [24], [25], [26]. A recent work [11] on semantic nighttime segmentation shows that images captured at twilight are helpful for supervision transfer from daytime to nighttime.…”

PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving

“…But complex environments, whether due to different lighting conditions, changing traffic flow, hazardous road conditions, complex vegetation, unexpected human movements or positions, or unfamiliar objects are all potential problems in real-world driving scenarios [4]. Datasets that capture richer and more diverse scenes, or provide different levels of annotation, can help improve the robustness of autonomous vehicles [5]. However, due to the impact of COVID-19, a large number of autonomous driving companies had to suspend their road testing in 2020, which led to a significant reduction of road test data.…”