Wasserstein Distance Guided Representation Learning for Domain Adaptation

Shen, Jian; Qu, Yanru; Zhang, Weinan; Yu, Yong

doi:10.48550/arxiv.1707.01217

Cited by 123 publications

(68 citation statements)

References 2 publications

(3 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In practice, transfer distances are often given by fdivergences [24], such as KL-divergence or the Hellinger distance, Wasserstein distances [25], and maximum mean discrepancy [18], [26], [27]. Others use generative adversarial networks, a deep learning distribution modeling technique, to estimate divergence [28], [29].…”

Section: B Behavioral Considerationsmentioning

confidence: 99%

A Systems Theory of Transfer Learning

Cody¹,

Beling²

2021

Preprint

View full text Add to dashboard Cite

Existing frameworks for transfer learning are incomplete from a systems theoretic perspective. They place emphasis on notions of domain and task, and neglect notions of structure and behavior. In doing so, they limit the extent to which formalism can be carried through into the elaboration of their frameworks. Herein, we use Mesarovician systems theory to define transfer learning as a relation on sets and subsequently characterize the general nature of transfer learning as a mathematical construct. We interpret existing frameworks in terms of ours and go beyond existing frameworks to define notions of transferability, transfer roughness, and transfer distance. Importantly, despite its formalism, our framework avoids the detailed mathematics of learning theory or machine learning solution methods without excluding their consideration. As such, we provide a formal, general systems framework for modeling transfer learning that offers a rigorous foundation for system design and analysis.

show abstract

Section: B Behavioral Considerationsmentioning

confidence: 99%

A Systems Theory of Transfer Learning

Cody¹,

Beling²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Transfer distance is usually referred to informally, e.g., to describe near or far transfer. It is implicit in the use of Wasserstein distance [37], maximum mean discrepancy [38], [39], generative adversarial networks [40], [41], and others, to calculate distributional-divergence-based components of loss functions in transfer learning algorithms. We consider transfer distance explicitly, in a way that may not necessarily be useful in calculating loss functions, but is interpretable to system designers and operators.…”

Section: Methodsmentioning

confidence: 99%

Empirically Measuring Transfer Distance for System Design and Operation

Cody,

Adams,

Beling

2021

Preprint

View full text Add to dashboard Cite

Classical machine learning approaches are sensitive to non-stationarity. Transfer learning can address nonstationarity by sharing knowledge from one system to another, however, in areas like machine prognostics and defense, data is fundamentally limited. Therefore, transfer learning algorithms have little, if any, examples from which to learn. Herein, we suggest that these constraints on algorithmic learning can be addressed by systems engineering. We formally define transfer distance in general terms and demonstrate its use in empirically quantifying the transferability of models. We consider the use of transfer distance in the design of machine rebuild procedures to allow for transferable prognostic models. We also consider the use of transfer distance in predicting operational performance in computer vision. Practitioners can use the presented methodology to design and operate systems with consideration for the learning theoretic challenges faced by component learning systems.

show abstract

“…of source and target features. After Generative Adversarial Network [13] was proposed, more works [57,6,15,58,45,59] leverage a domain discriminator to encourage domain confusion by an adversarial objective. Recently, image translation methods [60,14] have been adopted to further improve domain adaptation by performing domain alignment at pixel-level [15,11,16,61,62,17,63].…”

Section: Appendix a Additional Dataset Detailsmentioning

confidence: 99%

Multi-source Few-shot Domain Adaptation

Yue,

Zheng,

Reed

et al. 2021

Preprint

View full text Add to dashboard Cite

Multi-source Domain Adaptation (MDA) aims to transfer predictive models from multiple, fully-labeled source domains to an unlabeled target domain. However, in many applications, relevant labeled source datasets may not be available, and collecting source labels can be as expensive as labeling the target data itself. In this paper, we investigate Multi-source Few-shot Domain Adaptation (MFDA): a new domain adaptation scenario with limited multi-source labels and unlabeled target data. As we show, existing methods often fail to learn discriminative features for both source and target domains in the MFDA setting. Therefore, we propose a novel framework, termed Multi-Source Few-shot Adaptation Network (MSFAN), which can be trained end-to-end in a non-adversarial manner. MSFAN operates by first using a type of prototypical, multi-domain, self-supervised learning to learn features that are not only domain-invariant but also class-discriminative. Second, MSFAN uses a small, labeled support set to enforce feature consistency and domain invariance across domains. Finally, prototypes from multiple sources are leveraged to learn better classifiers. Compared with state-of-the-art MDA methods, MSFAN improves the mean classification accuracy over different domain pairs on MFDA by 20.2%, 9.4%, and 16.2% on Office, Office-Home, and DomainNet, respectively. * Equal contribution; work done when Zangwei was in Nanjing University.Preprint. Under review.

show abstract

Wasserstein Distance Guided Representation Learning for Domain Adaptation

Cited by 123 publications

References 2 publications

A Systems Theory of Transfer Learning

A Systems Theory of Transfer Learning

Empirically Measuring Transfer Distance for System Design and Operation

Multi-source Few-shot Domain Adaptation

Contact Info

Product

Resources

About