mDALU: Multi-Source Domain Adaptation and Label Unification with Partial Datasets

Abstract

One challenge of object recognition is to generalize to new domains, to more classes and/or to new modalities. This necessitates methods to combine and reuse existing datasets that may belong to different domains, have partial annotations, and/or have different data modalities. This paper formulates this as a multi-source domain adaptation and label unification problem, and proposes a novel method for it. Our method consists of a partially-supervised adaptation stage and a fully-supervised adaptation stage. In the former, partial knowledge is transferred from multiple source domains to the target domain and fused therein. Negative transfer between unmatching label spaces is mitigated via three new modules: domain attention, uncertainty maximization and attention-guided adversarial alignment. In the latter, knowledge is transferred in the unified label space after a label completion process with pseudo-labels. Extensive experiments on three different tasks -image classification, 2D semantic image segmentation, and joint 2D-3D semantic segmentation -show that our method outperforms all competing methods significantly.

Semantic Image Segmentation: Two Decades of Research

“…Finally, the segmentation models associated with different sources collaborate with each other to generate more reliable pseudo-labels for the target domain, used to refine the models. Gong et al (2021b) consider the case where the aim is to learn from different source datasets with potentially different class sets, and formulate the task as a multi-source domain adaptation with label unification. To approach this, they propose a two-step solution: first, the knowledge is transferred form the multiple sources to the target; second, a unified label space is created by exploiting pseudo-labels, and the knowledge is further transferred to this representation space.…”

“…For what concerns segmentation, Gong et al (2021b) propose an MSDA strategy where the label space of the target domain is defined as the union of the label spaces of all the different source domains and the knowledge in different label spaces is transferred from different source domains to the target domain, where the missing labels are replaced by pseudo-labels. Liu et al (2021a) propose an optimization scheme which alternates be-tween 1) conditional distribution alignment with adversarial UDA relying on estimated class-wise balancing in the target and 2) target label proportion estimates with Mean Matching (Gretton et al, 2009), assuming conditional distributions alignment between the domains.…”

Unsupervised Domain Adaptation for Semantic Image Segmentation: a Comprehensive Survey

“…Gong et al [58] consider the case where the aim is to learn from different source datasets with potentially different class sets, and formulates it as a multi-source domain adaptation with "label unification". To approach this, they propose a two-step solution: first, the knowledge is transferred form the multiple sources to the target; second, a unified label space is created by exploiting pseudo-labels, and the knowledge is further transferred to this representation space.…”

“…For what concerns segmentation, Gong et al [58] propose a multi-source domain adaptation strategy where the label space of the target domain is defined as the union of the label spaces of all the different source domains and the knowledge in different label spaces is transferred from different source domains to the target domain and where the missing labels are replaced by pseudo-labels.…”