SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation

Abstract

Domain adaptive semantic segmentation attempts to make satisfactory dense predictions on an unlabeled target domain by utilizing the supervised model trained on a labeled source domain. One popular solution is self-training, which retrains the model with pseudo labels on target instances. Plenty of methods tend to alleviate noisy pseudo labels, however, they ignore intrinsic connections among cross-domain pixels with similar semantic concepts. In consequence, they would struggle to deal with the semantic variations across domains and to build a category-discriminative embedding space, leading to less discrimination and poor generalization. In this work, we propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixel to promote learning of class-discriminative and class-balanced pixel embedding space across domains, eventually boosting the performance of self-training methods. Specifically, to explore proper semantic concepts, we first investigate a centroid-aware pixel contrast that employs the category centroids of the entire source domain or a single source image to guide the learning of discriminative features. Considering the possible lack of category diversity in semantic concepts, we then blaze a trail of distributional perspective to involve a sufficient quantity of instances, namely distribution-aware pixel contrast, in which we approximate the true distribution of each semantic category from the statistics of labeled source data. Moreover, such an optimization objective can derive a closed-form upper bound by implicitly involving an infinite number of (dis)similar pairs, making it computationally efficient. Extensive experiments show that SePiCo not only helps stabilize training but also yields discriminative features, making significant progress on both synthetic-to-real and daytime-to-nighttime scenarios. Most notably, SePiCo establishes state-of-the-art performance on tasks of GTAV → Cityscapes, SYNTHIA → Cityscapes and Cityscapes → Dark Zurich, improving by 12.8% mIoU, 8.8% mIoU, and 9.2% mIoU compared to the previous best method, respectively. The code will be available at https://github.com/BIT-DA/SePiCo.

MSCDA: Multi-level Semantic-guided Contrast Improves Unsupervised Domain Adaptation for Breast MRI Segmentation in Small Datasets

“…DART (Shanis et al (2019)) and MT-UDA (Zhao et al (2021b)) combined self-training with adversarial learning in different ways, both receiving promising results. For imbalanced datasets, different denoising methods and sampling strategies have been proposed to improve the quality of pseudo-labels (Zhang et al (2021);Hoyer et al (2022); Xie et al (2022)). Similar to Chaitanya et al (2020), recent self-training approaches (Xie et al (2022); Zhang et al (2022)) incorporated CL, i.e.…”

“…For imbalanced datasets, different denoising methods and sampling strategies have been proposed to improve the quality of pseudo-labels (Zhang et al (2021);Hoyer et al (2022); Xie et al (2022)). Similar to Chaitanya et al (2020), recent self-training approaches (Xie et al (2022); Zhang et al (2022)) incorporated CL, i.e. unsupervised contrastive domain adaptation, to align cross-domain features by sampling or merging contrastive feature embeddings across categories.…”

“…For P2C contrast, we force the pixel representation to learn a more general representation with the guidance of the centroid (Wang et al (2021); Xie et al (2022)). Specifically, a pixel and a centroid from the same category are considered as a positive pair (v k , c k+ ), while a pixel and a centroid from different categories are considered as a negative pair (v k , c q− ).…”

Iterative Loop Method Combining Active and Semi-Supervised Learning for Domain Adaptive Semantic Segmentation

“…Nowadays, many research works have proved that domain adaptation is one of the powerful means to address the above issues [23,37,44,60,63]. Among them, unsupervised domain adaptation (UDA) [26,28,33,41,69,77] aims to solve this problem by leveraging the knowledge of labelrich data (source data) and transferring it to unlabeled data (target data) [52]. While it can avoid the intensive workload of manual annotation, the performance still lags far behind fully supervised models [56].…”

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

“…Domain adaptation has been thoroughly studied to improve model generalization to unseen domains, e.g., adapting to the real world from synthetic data collections [81], [82]. Two predominant categories of unsupervised domain adaptation fall either in self-training [83], [84], [85], [86], [87], [88] or adversarial learning [89], [90], [91]. Self-training methods usually generate pseudo-labels to gradually adapt through iterative improvement [92], whereas adversarial solutions build on the idea of GANs [93] to conduct image translation [89], [94], or enforce alignment in layout matching [95], [96] and feature agreement [3], [16].…”