Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

Wang, Qin; Dai, Dengxin; Hoyer, Lukas; Gool, Luc Van; Fink, Olga

doi:10.1109/iccv48922.2021.00840

Cited by 95 publications

(56 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [25] proposes to refine pseudo-labels by making use of prototypes. Wang et al [26] find that explicitly learning the relationship between the main task and selfsupervised auxiliary task can help improve DA performance. Mei et al [27] find it beneficial for DA to integrate adversarial alignment with self-training.…”

Section: B Domain Adaptationmentioning

confidence: 99%

Integrating Expert Knowledge With Domain Adaptation for Unsupervised Fault Diagnosis

Wang

Taal

Fink

2022

IEEE Trans. Instrum. Meas.

Self Cite

View full text Add to dashboard Cite

Data-driven fault diagnosis methods often require abundant labeled examples for each fault type. On the contrary, real-world data is often unlabeled and consists of mostly healthy observations and only few samples of faulty conditions. The lack of labels and fault samples imposes a significant challenge for existing data-driven fault diagnosis methods. In this article, we aim to overcome this limitation by integrating expert knowledge with domain adaptation (DA) in a synthetic-to-real framework for unsupervised fault diagnosis. Motivated by the fact that domain experts often have a relatively good understanding on how different fault types affect healthy signals, in the first step of the proposed framework, a synthetic fault dataset is generated by augmenting real vibration samples of healthy bearings. This synthetic dataset integrates expert knowledge and encodes class information about the faults types. However, models trained solely based on the synthetic data often do not perform well because of the distinct distribution difference between the synthetically generated and real faults. To overcome this domain gap between the synthetic and real data, in the second step of the proposed framework, an imbalance-robust DA approach is proposed to adapt the model from synthetic faults (source) to the unlabeled real faults (target) which suffer from severe class imbalance. The framework is evaluated on two unsupervised fault diagnosis cases for bearings, the CWRU laboratory dataset and a real-world wind-turbine dataset. Experimental results demonstrate that the generated faults are effective for encoding fault type information and the DA is robust against the different levels of class imbalance between faults.

show abstract

Section: B Domain Adaptationmentioning

confidence: 99%

Integrating Expert Knowledge With Domain Adaptation for Unsupervised Fault Diagnosis

Wang

Taal

Fink

2022

IEEE Trans. Instrum. Meas.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, many works opt for a stage-wise training mechanism to avoid training error amplification in a single-stage model, which heavily relies on a warm-up stage to increase the reliability of the generated pseudo labels. Hereafter, several methods combine adversarial training and self-training [37], [38] or train with auxiliary tasks [39], [40] to learn discriminative representations from unlabeled target data. Contrastive learning is a relevant topic, which learns proper visual representations by comparing different unlabeled data [41], [42], [43], [44].…”

Section: Memory Bankmentioning

confidence: 99%

SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation

Xie¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Domain adaptive semantic segmentation attempts to make satisfactory dense predictions on an unlabeled target domain by utilizing the supervised model trained on a labeled source domain. One popular solution is self-training, which retrains the model with pseudo labels on target instances. Plenty of methods tend to alleviate noisy pseudo labels, however, they ignore intrinsic connections among cross-domain pixels with similar semantic concepts. In consequence, they would struggle to deal with the semantic variations across domains and to build a category-discriminative embedding space, leading to less discrimination and poor generalization. In this work, we propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixel to promote learning of class-discriminative and class-balanced pixel embedding space across domains, eventually boosting the performance of self-training methods. Specifically, to explore proper semantic concepts, we first investigate a centroid-aware pixel contrast that employs the category centroids of the entire source domain or a single source image to guide the learning of discriminative features. Considering the possible lack of category diversity in semantic concepts, we then blaze a trail of distributional perspective to involve a sufficient quantity of instances, namely distribution-aware pixel contrast, in which we approximate the true distribution of each semantic category from the statistics of labeled source data. Moreover, such an optimization objective can derive a closed-form upper bound by implicitly involving an infinite number of (dis)similar pairs, making it computationally efficient. Extensive experiments show that SePiCo not only helps stabilize training but also yields discriminative features, making significant progress on both synthetic-to-real and daytime-to-nighttime scenarios. Most notably, SePiCo establishes state-of-the-art performance on tasks of GTAV → Cityscapes, SYNTHIA → Cityscapes and Cityscapes → Dark Zurich, improving by 12.8% mIoU, 8.8% mIoU, and 9.2% mIoU compared to the previous best method, respectively. The code will be available at https://github.com/BIT-DA/SePiCo.

show abstract

“…Guizilini et al (2021) utilize multi-task learning of semantic segmentation and SDE to learn a more domain-invariant representation. Instead of applying the view synthesis loss from SDE directly, Wang et al (2021) use depth pseudo-labels from an SDE teacher network to learn depth estimation and semantic segmentation in a multi-tasking framework. To better transfer knowledge between both domains and tasks, the correlation of depth and semantic segmentation features is explicitly transferred from the source to the target domain and the depth adaptation difficulty is transferred to semantic segmentation to weigh the trust in the semantic segmentation pseudo-labels.…”

Section: Auxiliary Depth Estimation For Domain Adaptationmentioning

confidence: 99%

Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

Hoyer¹,

Dai²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Training deep networks for semantic segmentation requires large amounts of labeled training data, which presents a major challenge in practice, as labeling segmentation masks is a highly labor-intensive process. To address this issue, we present a framework for semi-supervised and domain-adaptive semantic segmentation, which is enhanced by self-supervised monocular depth estimation (SDE) trained only on unlabeled image sequences.In particular, we utilize SDE as an auxiliary task comprehensively across the entire learning framework: First, we automatically select the most useful samples to be annotated for semantic segmentation based on the correlation of sample diversity and difficulty between SDE and semantic segmentation. Second, we implement a strong data augmentation by mixing images and labels using the geometry of the scene. Third, we transfer knowledge from features learned during SDE to semantic segmentation by means of transfer and multi-task learning. And fourth, we exploit additional labeled synthetic data with Cross-Domain DepthMix and Matching Geometry Sampling to align synthetic and real data.

show abstract

Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

Cited by 95 publications

References 44 publications

Integrating Expert Knowledge With Domain Adaptation for Unsupervised Fault Diagnosis

Integrating Expert Knowledge With Domain Adaptation for Unsupervised Fault Diagnosis

SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation

Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

Contact Info

Product

Resources

About