Fully Convolutional Adaptation Networks for Semantic Segmentation

Abstract

The recent advances in deep neural networks have convincingly demonstrated high capability in learning vision models on large datasets. Nevertheless, collecting expert labeled datasets especially with pixel-level annotations is an extremely expensive process. An appealing alternative is to render synthetic data (e.g., computer games) and generate ground truth automatically. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. In this paper, we facilitate this issue from the perspectives of both visual appearance-level and representation-level domain adaptation. The former adapts source-domain images to appear as if drawn from the "style" in the target domain and the latter attempts to learn domain-invariant representations. Specifically, we present Fully Convolutional Adaptation Networks (FCAN), a novel deep architecture for semantic segmentation which combines Appearance Adaptation Networks (AAN) and Representation Adaptation Networks (RAN). AAN learns a transformation from one domain to the other in the pixel space and RAN is optimized in an adversarial learning manner to maximally fool the domain discriminator with the learnt source and target representations. Extensive experiments are conducted on the transfer from GTA5 (game videos) to Cityscapes (urban street scenes) on semantic segmentation and our proposal achieves superior results when comparing to state-of-theart unsupervised adaptation techniques. More remarkably, we obtain a new record: mIoU of 47.5% on BDDS (drivecam videos) in an unsupervised setting.

Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation

“…Thus, we consider that performing both alignment aspects in one unified framework can leverage their individual advantages to improve domain adaptation performance. Importantly, the combination between image and feature alignment should be a synergistic merge to exploit their mutual interactions and benefits, which has not been tapped in previous works [18], [19].…”

“…Combining these two adaptive strategies to achieve a stronger domain adaption technique is under explorable progress. As the state-of-the-art methods for semantic segmentation adaptation methods, CyCADA [18] and Zhang et al [19] achieved leading performance in adaptation between synthetic to real world driving scene domains. However, their image and feature alignments are sequentially connected and trained in separate stages without interactions.…”

Constructing Self-Motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach

“…We follow the experimental setup of previous works [38,4,40,43] and use the standard benchmark settings (i.e., "GTAV to Cityscapes" and "SYNTHIA to Cityscapes") in the experiments.…”

Customizable Architecture Search for Semantic Segmentation

“…It enables spatial dense prediction and efficient end-to-end training. Following FCN, researchers propose several advanced techniques ranging from cross-layer feature ensemble [10,15,24,32] to context information exploita-tion [4,5,6,18,21,27,35,37]. The FCN formulation could be further improved by employing post-processing techniques, such as the conditional random field [4], to consider label spatial consistency.…”