A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes

Abstract

During the last half decade, convolutional neural networks (CNNs) have triumphed over semantic segmentation, which is one of the core tasks in many applications such as autonomous driving and augmented reality. However, to train CNNs requires a considerable amount of data, which is difficult to collect and laborious to annotate. Recent advances in computer graphics make it possible to train CNNs on photo-realistic synthetic imagery with computer-generated annotations. Despite this, the domain mismatch between the real images and the synthetic data hinders the models' performance. Hence, we propose a curriculum-style learning approach to minimizing the domain gap in urban scene semantic segmentation. The curriculum domain adaptation solves easy tasks first to infer necessary properties about the target domain; in particular, the first task is to learn global label distributions over images and local distributions over landmark superpixels. These are easy to estimate because images of urban scenes have strong idiosyncrasies (e.g., the size and spatial relations of buildings, streets, cars, etc.). We then train a segmentation network, while regularizing its predictions in the target domain to follow those inferred properties. In experiments, our method outperforms the baselines on two datasets and two backbone networks. We also report extensive ablation studies about our approach.

Constructing Self-Motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach

“…One is based on a curriculum learning strategy. Zhang et al first learns to solve easy tasks in the target domain and then use them to regularize semantic segmentation [39,38]. Dai et al construct a curriculum by simulating foggy images of different fog densities [7].…”

“…In this section, we first reveal the connection between the curriculum domain adaptation (CDA) [38] and the selftraining (ST) for adaptation [43]. This connection naturally leads to the training algorithm of this paper, dubbed self-motivated pyramid curriculum domain adaptation (Py-CDA), for the semantic segmentation task.…”

“…In order to better take advantage of the synthetic imagery for the semantic segmentation of real scenes, we pro-arXiv:1908.09547v1 [cs.CV] 26 Aug 2019 pose a novel domain adaptation approach by drawing an intriguing connection between two previous works: curriculum domain adaptation [39,38] and self-training [43]. This connection naturally leads to self-motivated pyramid curriculums and a new training algorithm for the cross-domain adaptation of semantic segmentation networks.…”

“…Our main contribution is two-fold. One is that we pro-vide a new insight connecting the self-training for adapting segmentation networks [43] and the curriculum adaptation method [39,38]. The other is that, inspired by the connection, we propose a novel self-motivated pyramid curriculum for the domain adaptation of semantic segmentation networks.…”

“…The other is that, inspired by the connection, we propose a novel self-motivated pyramid curriculum for the domain adaptation of semantic segmentation networks. Extensive experiments show that it outperforms either [43] or [38] individually. Moreover, it is on par with or better than state-of-the-art adversarial adaptation methods without the need of maintaining an extra discrminator network or carefully tuning the optimization procedure for min max problems.…”