Framework for Generation and Removal of Multiple Types of Adverse Weather from Driving Scene Images

Abstract

Weather variation in the distribution of image data can cause a decline in the performance of existing visual algorithms during evaluation. Adding additional samples of target domain to training data or using pre-trained image restoration methods such as de-hazing, de-raining, and de-snowing, to improve the quality of input images are two promising solutions. In this work, we propose Multiple Weather Translation GAN (MWTG), a CycleGAN-based, dual-purpose framework that simultaneously learns weather generation and its removal from image data. MWTG consists of four GANs constrained using cycle consistency that carry out domain translation tasks between hazy, rainy, snowy, and clear weather, using an asymmetric approach. To increase network capacity, we employ a spatial feature transform (SFT) layer to fuse the features extracted from the weather layer, which contains high-level domain information from the previous generators. Further, we collect an unpaired, real-world driving dataset recorded under various weather conditions called Realistic Driving Scenes under Bad Weather (RDSBW). We qualitatively and quantitatively evaluate MWTG using the RDSBW and the variation of Cityscapes that synthesize weather effects, eg., FoggyCityscape. Our experimental results suggest that MWTG can generate realistic weather in clear images and also accurately remove noise from weather images. Furthermore, the SOTA pedestrian detector ASCP is shown to achieve an impressive gain in detection precision after image restoration using the proposed MWTG method.

Controllable Unsupervised Snow Synthesis by Latent Style Space Manipulation

“…The goal is to learn the mapping between the snow domain and the clear weather domain. In our previous unpaired I2I methods [29,30], two generators are employed to transfer images into the expected domain. Two corresponding discriminators are employed to differentiate real images and fake images.…”

“…Some of the examples are shown in the Figure 4. As a result, our curated snow dataset [30] comprises a total of 6814 meticulously curated photographs. The number of intercepted images we keep is the same as for cityscapes because GAN training is prone to problems such as mode collapse, which leads to training failure.…”

L-DIG: A GAN-Based Method for LiDAR Point Cloud Processing under Snow Driving Conditions

“…The development of robust weather models benefits from training on paired data, i.e., a pair of weather-corrupted data and clear data with the rest of the elements identical, which are commonly obtained via artificially synthesizing realistic weather effects in previously clear driving scene images [10][11][12]. Such an approach has been proven highly effective in rain [13,14], fog [15,16], and snow [17] weather conditions in camera images, plus contaminations on the camera lens [18]. However, due to the relatively low data density, the realization of the weather effects in point clouds still largely depends on the collections in weather chambers [19,20] before the mature realization of weather data augmentation in point clouds.…”

“…Therefore, it is necessary to develop a way to work with few paired data or unpaired data. In terms of disentangled data processing, CycleGAN [22] demonstrates a high ability in style conversion and object generation [23] based on datasets with different backgrounds and from different domains, and its implementation in weather models has been proven feasible [17,24]. In this research, we propose the 'L-DIG' (LiDAR depth images GAN), a GAN-based method using depth image priors for LiDAR point cloud processing under various snow conditions.…”