DCAN: Dual Channel-Wise Alignment Networks for Unsupervised Scene Adaptation

Abstract

Harvesting dense pixel-level annotations to train deep neural networks for semantic segmentation is extremely expensive and unwieldy at scale. While learning from synthetic data where labels are readily available sounds promising, performance degrades significantly when testing on novel realistic data due to domain discrepancies. We present Dual Channel-wise Alignment Networks (DCAN), a simple yet effective approach to reduce domain shift at both pixel-level and feature-level. Exploring statistics in each channel of CNN feature maps, our framework performs channel-wise feature alignment, which preserves spatial structures and semantic information, in both an image generator and a segmentation network. In particular, given an image from the source domain and unlabeled samples from the target domain, the generator synthesizes new images on-the-fly to resemble samples from the target domain in appearance and the segmentation network further refines highlevel features before predicting semantic maps, both of which leverage feature statistics of sampled images from the target domain. Unlike much recent and concurrent work relying on adversarial training, our framework is lightweight and easy to train. Extensive experiments on adapting models trained on synthetic segmentation benchmarks to real urban scenes demonstrate the effectiveness of the proposed framework.

Bidirectional Learning for Domain Adaptation of Semantic Segmentation

“…Recently, domain adaptation for semantic segmentation has made good progress by separating it into two sequential steps. It firstly translates images from the source domain to the target domain with an image-to-image translation model (e.g., CycleGAN [38]) and then add a discriminator on top of the features of the segmentation model to further decrease the domain gap [12,36]. When the domain gap is reduced by the former step, the latter one is easy to learn and can further decrease the domain shift.…”

“…In this paper, we propose a new bidirectional learning framework for domain adaptation of image semantic segmentation. The system involves two separated modules: image-to-image translation model and segmentation adaptation model similar to [12,36], but the learning process involves two directions (i.e., "translation-to-segmentation" and "segmentation-to-translation"). The whole system forms a closed-loop learning.…”

“…Experiments show that our method is superior to the state-of-the-art methods in domain adaptation of segmentation with a big margin. The source code is available at https://github.com/liyunsheng13/BDL.On the forward direction (i.e., "translation-tosegmentation", similar to [12,36]), we propose a self-supervised learning (SSL) approach in training our segmentation adaptation model. Different from segmentation models trained on real data, the segmentation adaptation model is trained on both synthetic and real datasets, but the real data has no annotations.…”

“…On the forward direction (i.e., "translation-tosegmentation", similar to [12,36]), we propose a self-supervised learning (SSL) approach in training our segmentation adaptation model. Different from segmentation models trained on real data, the segmentation adaptation model is trained on both synthetic and real datasets, but the real data has no annotations.…”

Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation

“…We report experimental results of the proposed method on two adaptation experiments in Table 1. We compare our proposed method with Curriculum DA [51], CyCADA [14], MCD [39], LSD-seg [41], AdaptSegNet [44], ROAD [5], Conservative Loss [55], DCAN [47], and CBST [57]. In Table 1, Self-Ensembling (SE) represents the segmentation performance of the network trained by source and target through the self-ensembling, without our data augmentation method.…”