Multi-weather city: Adverse weather stacking for autonomous driving

Abstract

Autonomous vehicles make use of sensors to perceive the world around them, with heavy reliance on visionbased sensors such as RGB cameras. Unfortunately, since these sensors are affected by adverse weather, perception pipelines require extensive training on visual data under harsh conditions in order to improve the robustness of downstream tasks -data that is difficult and expensive to acquire. Based on GAN and CycleGAN architectures, we propose an overall (modular) architecture for constructing datasets, which allows one to add, swap out and combine components in order to generate images with diverse weather conditions. Starting from a single dataset with ground-truth, we generate 7 versions of the same data in diverse weather, and propose an extension to augment the generated conditions, thus resulting in a total of 14 adverse weather conditions, requiring a single ground truth. We test the quality of the generated conditions both in terms of perceptual quality and suitability for training downstream tasks, using real world, out-of-distribution adverse weather extracted from various datasets. We show improvements in both object detection and instance segmentation across all conditions, in many cases exceeding 10 percentage points increase in AP, and provide the materials and instructions needed to re-construct the multi-weather dataset, based upon the original Cityscapes dataset.

YOLO-Z: Improving small object detection in YOLOv5 for autonomous vehicles

“…Its original purpose is to help plan a path for an autonomous racing vehicle based on the colours of the cones, knowing that there are a total of 4 classes (yellow, blue, orange and big orange cones) and close to 4,000 images (see Figure 1, 2). This dataset includes digitally augmented images [18] and cases with challenging weather conditions. A dataset such as this one models more complex tasks in autonomous vehicles.…”

Vision in adverse weather: Augmentation using CycleGANs with various object detectors for robust perception in autonomous racing

“…Other articles have used CycleGAN in the context of autonomous driving tasks [33,47,29]. The latter is the closest to our work; using CycleGAN to create fake adverse conditions for road-going AVs in general rather than racing AVs.…”

“…A recent promising approach is harnessing CycleGAN to generate more realistic weather augmentations using style transfer. A small number of studies have explored the use of CycleGAN to improve perception performance in adverse weather in the context of on-road autonomous driving tasks [33,47,29], but as yet, no studies have attempted to exploit this potential in the context of autonomous racing. Furthermore, various object detection models have been proposed for real-time use in AV applications [19,20,38,44], however few studies have compared different detectors for use in autonomous driving applications [15] -let alone with a detailed comparison of the detection performance and latencies introduced.…”

RestoreX-AI: A Contrastive Approach towards Guiding Image Restoration via Explainable AI Systems

“…The thermal variations accompanying weather change can adversely impact the optical, electronic, and mechanical components used in capturing visual data, thus harming the performance of visual recognition systems [6]. Frigid temperatures, snowfall or dense fog, for example, can cause condensation on the lens, further blurring the view and obscuring the object boundaries; rain streaks on car windows can generate glares or act as a double lens [20]. For an autonomous car, it is critical and essential to overcome the effects of weather conditions to ensure reliability.…”