Learning Semantic Segmentation From Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach

Abstract

Recently, increasing attention has been drawn to training semantic segmentation models using synthetic data and computer-generated annotation. However, domain gap remains a major barrier and prevents models learned from synthetic data from generalizing well to real-world applications. In this work, we take the advantage of additional geometric information from synthetic data, a powerful yet largely neglected cue, to bridge the domain gap. Such geometric information can be generated easily from synthetic data, and is proven to be closely coupled with semantic information. With the geometric information, we propose a model to reduce domain shift on two levels: on the input level, we augment the traditional image translation network with the additional geometric information to translate synthetic images into realistic styles; on the output level, we build a task network which simultaneously performs depth estimation and semantic segmentation on the synthetic data. Meanwhile, we encourage the network to preserve the correlation between depth and semantics by adversarial training on the output space. We then validate our method on two pairs of synthetic to real dataset: Virtual KITTI→KITTI, and SYNTHIA→Cityscapes, where we achieve a significant performance gain compared to the non-adaptive baseline and methods without using geometric information. This demonstrates the usefulness of geometric information from synthetic data for cross-domain semantic segmentation.

Significance-Aware Information Bottleneck for Domain Adaptive Semantic Segmentation

“…Joint consideration of pixel-and feature-space domain adaptation is studied in [13]. For segmentation task, it is also found that aligning the segmentation space is a more effective DA strategy [44,7]. Besides the adversarial training-based DA methods [13,44,20], other lines of work on semantic segmentation borrow the idea from selftraining [33] or co-training [47].…”

Self-Supervised Learning With Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera

“…Several methods train networks based on ground-truth labels and have been successfully applied to many tasks, such as monocular depth prediction [9,10,27,28], optical flow [8,19,35,38] and camera pose estimation [22,23,24]. To leverage the multiple cues, the tasks can be tackled jointly by fusing boundary detection [20], the estimation of surface normals [28], semantic segmentation [4,46,50], etc.…”

Adversarial Learning and Self-Teaching Techniques for Domain Adaptation in Semantic Segmentation

“…A curriculum-style learning approach is proposed in [16], where firstly the easier task of estimating global label distributions is learned and then the segmentation network is trained forcing that the target label distribution is aligned to the previously computed properties. Many other works addressed the domain adaptation problem with various techniques, including GANs [29], [45], cycle consistency [11], [46], output space alignment [47], [48], distillation loss [17], [49], class-balanced selftraining [50], conservative loss [51], geometrical guidance [52], adaptation networks [53] and entropy minimization [54].…”