Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding

Abstract

This work addresses the problem of semantic scene understanding under fog. Although marked progress has been made in semantic scene understanding, it is mainly concentrated on clear-weather scenes. Extending semantic segmentation methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both labeled synthetic foggy data and unlabeled real foggy data. The method is based on the fact that the results of semantic segmentation in moderately adverse conditions (light fog) can be bootstrapped to solve the same problem in highly adverse conditions (dense fog). CMAda is extensible to other adverse conditions and provides a new paradigm for learning with synthetic data and unlabeled real data. In addition, we present four other main stand-alone contributions: 1) a novel method to add synthetic fog to real, clear-weather scenes using semantic input; 2) a new fog density estimator; 3) a novel fog densification method for real foggy scenes without known depth; and 4) the Foggy Zurich dataset comprising 3808 real foggy images, with pixel-level semantic annotations for 40 images with dense fog. Our experiments show that 1) our fog simulation and fog density estimator outperform their state-of-the-art counterparts with respect to the task of semantic foggy scene understanding (SFSU); 2) CMAda improves the performance of state-ofthe-art models for SFSU significantly, benefiting both from our synthetic and real foggy data. The foggy datasets and code are publicly available.

Semantic Understanding of Foggy Scenes with Purely Synthetic Data

“…Initially, its test split Foggy Zurich-test consisted of 16 images with pixel-level semantic annotations for 18 out of 19 evaluation classes of Cityscapes [8] (the train class is missing). In a more recent work [42], Foggy Zurich-test has been extended and now includes pixel-level semantic annotations for in total 40 images. These 40 images form the test set that we used for our evaluation.…”

Constructing Self-Motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach

“…Zhang et al first learns to solve easy tasks in the target domain and then use them to regularize semantic segmentation [39,38]. Dai et al construct a curriculum by simulating foggy images of different fog densities [7]. In this work, we also view the self-training [43] as a curriculum-style domain adaptation method.…”

Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

“…denotes the predicted labels of image I z−1 n . A simple way to get these labels is by directly feeding I z−1 n to φ z−1 , similar to the approach of [7,26] for the case of fog. This choice, however, suffers from accumulation of substantial errors in the prediction of φ z−1 into the subsequent training step if domain z − 1 is not the daytime domain.…”