Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding

Abstract

This work addresses the problem of semantic scene understanding under dense fog. Although considerable progress has been made in semantic scene understanding, it is mainly related to clear-weather scenes. Extending recognition methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both synthetic and real foggy data. In addition, we present three other main stand-alone contributions: 1) a novel method to add synthetic fog to real, clear-weather scenes using semantic input; 2) a new fog density estimator; 3) the Foggy Zurich dataset comprising 3808 real foggy images, with pixel-level semantic annotations for 16 images with dense fog. Our experiments show that 1) our fog simulation slightly outperforms a state-of-the-art competing simulation with respect to the task of semantic foggy scene understanding (SFSU); 2) CMAda improves the performance of state-of-the-art models for SFSU significantly by leveraging unlabeled real foggy data. The datasets and code will be made publicly available.

Semantic Understanding of Foggy Scenes with Purely Synthetic Data

“…In contrast to prior work [3], [21] only using real input images (with incomplete and imperfect depth information) to the synthetic fog generation pipeline, our work focuses on synthetic input images (with complete and perfect depth information). Hence, our resulting images are purely synthetic.…”

“…The fog simulation pipeline of [3] is improved in [21] by leveraging semantic annotations to increase the accuracy of the required depth map, resulting in the Foggy Cityscapes-DBF dataset. Both of these synthetic foggy datasets have been utilized in [3], [21] to improve semantic segmentation performance of state-of-the-art CNN models [14], [15] on real foggy benchmarks. We are inspired by both lines of research and combine the fully controlled setting of purely synthetic data with the synthetic fog generation pipeline of [3], [21].…”

CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency

“…Learning from synthetic data. Training the model on large-scale synthetic datasets has been extensively studied in semantic segmentation [44,45,17,16,9,19,37,38,56], multi-view stereo [20], depth estimation [51], optical flow [43,21,23], amodal segmentation [18], and object detection [9,33]. In our work, we show that the proposed cross-domain consistency loss can be applied not only to synthetic-to-real adaptation but to real-to-real adaptation tasks as well.…”

Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

“…denotes the predicted labels of image I z−1 n . A simple way to get these labels is by directly feeding I z−1 n to φ z−1 , similar to the approach of [7,26] for the case of fog. This choice, however, suffers from accumulation of substantial errors in the prediction of φ z−1 into the subsequent training step if domain z − 1 is not the daytime domain.…”

“…This enables adding intermediate domains between the two to smoothly transfer semantic knowledge. This idea is found to be effective in [8,26]; we extend it by adding two more modules. Place: images taken over different time but with the same 6D camera pose share a large portion of content.…”