Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization Without Accessing Target Domain Data

Abstract

We propose to harness the potential of simulation for the semantic segmentation of real-world self-driving scenes in a domain generalization fashion. The segmentation network is trained without any data of target domains and tested on the unseen target domains. To this end, we propose a new approach of domain randomization and pyramid consistency to learn a model with high generalizability. First, we propose to randomize the synthetic images with the styles of real images in terms of visual appearances using auxiliary datasets, in order to effectively learn domain-invariant representations. Second, we further enforce pyramid consistency across different "stylized" images and within an image, in order to learn domaininvariant and scale-invariant features, respectively. Extensive experiments are conducted on the generalization from GTA and SYNTHIA to Cityscapes, BDDS and Mapillary; and our method achieves superior results over the stateof-the-art techniques. Remarkably, our generalization results are on par with or even better than those obtained by state-of-the-art simulation-to-real domain adaptation methods, which access the target domain data at training time.

Deep Domain-Adversarial Image Generation for Domain Generalisation

“…In this paper, we tackle the DG problem by synthesising data from unseen domains. We assume that augmenting the original training data of source domains with synthetic data from unseen domains could make the task model intrinsically more domain-generalisable (Tobin et al 2017;Yue et al 2019). To this end, a novel framework based on Deep Domain-Adversarial Image Generation (DDAIG) is introduced, which is illustrated in Figure 1.…”

Foreground-Aware Stylization and Consensus Pseudo-Labeling for Domain Adaptation of First-Person Hand Segmentation

“…When the test data is not available during training due to some restriction, e.g., pertaining to resources or privacy issues, generalizing to the test domain is important. We experimented with our method in a multi-auxiliary domain generalization setting [62]. Here, we chose a domain from the target domains as a test domain.…”

“…When we cannot access the target data due to privacy issues [45], [51], [58], generalizing to an unseen domain is also necessary. We also experimented on a multitarget domain adaptation setting [2], [11], which aims to simultaneously adapt to multiple target domains, and a multiauxiliary domain generalization setting [60], which aims to generalize to an unseen test domain utilizing knowledge from auxiliary domains.…”

Meta-Learned Feature Critics for Domain Generalized Semantic Segmentation

“…Domain generalization for semantic segmentation has seen less progress due to its task difficulty (pixel-wise classification) compared to classification. Recent works are modelbased [10], embedding domain generalization capability into the model by modifying the model structure, or data-based [11], augmenting source domain data with styles from additional datasets (e.g., ImageNet [12]). Although yielding positive results, their performance depend heavily on the style variance of the additional data introduced.…”