Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency

Abstract

Image-to-image translation has recently received significant attention due to advances in deep learning. Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way. However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner-and cross-domain variations. To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain. We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain. Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain. To avoid semantic inconsistencies during translation that naturally appear due to the large inner-and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels. Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening

“…Semantic segmentation in DG Based on the synthetic data such as GTAV [47] and SYNTHIA [49], numerous DA studies [43,60,53,6,69,18,58,37,65] have been proposed in semantic segmentation, but only a few DG studies [64,44] address semantic segmentation, as the majority of the DG methods mainly focused on image classification. DA, which can access the target domains, generally has better performance than DG, but DG methods that can handle an arbitrary unseen domain without access to the target domain are mandatory in the real world.…”

Discover, Hallucinate, and Adapt: Open Compound Domain Adaptation for Semantic Segmentation

“…To do so, we collect domain-specific representations of each target image and assign pseudo domain labels by clustering (i.e., k-means clustering [17]). In this work, we assume that the latent domain of images is reflected in their style [16,19,27,15,4,37], and we thus use style information to cluster the compound target domain. In practice, we introduce hyperparameter K and divide the compound target domain T into a total of K latent domains by style, {T j } K j=1 .…”

“…In particular, the scheme starts by discovering K latent domains in the compound target data [29] (discover). Motivated by the previous works [16,19,27,15,4,37] that utilizes style information as domain-specific representation, we propose to use latent target styles to cluster the compound target. Then, the scheme generates K target-like source domains by adopting an examplar-guided image translation network [5,42], hallucinating multiple latent target domains in source (hallucinate).…”

Semantic Map Injected GAN Training for Image-to-Image Translation

“…SemGAN includes an additional loss named consistency constraints along with existing GAN loss and cycle consistency loss. Exemplar guided unsupervised image to image translation with semantic consistency [21] uses feature masks to avoid the semantic inconsistency. This allows to transfer style information of the target image.…”