A Survey of Unsupervised Deep Domain Adaptation

Wilson, Garrett; Cook, Diane J.

doi:10.48550/arxiv.1812.02849

Cited by 13 publications

(18 citation statements)

References 143 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a large body of work addressing covariate shift issues in the literature on domain adaptation. Wilson and Cook [2018] survey different methods in unsupervised domain adaptation. These include methods that learn mappings between domains [Fernando et al, 2013, Sener et al, 2016, match means and covariances across feature vectors , or match moments of the distributions directly [Peng et al, 2019] or through kernel embeddings [Long et al, 2015, Gong et al, 2012.…”

Section: Related Workmentioning

confidence: 99%

Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift

Nado,

Padhy,

Sculley

et al. 2020

Preprint

View full text Add to dashboard Cite

Covariate shift has been shown to sharply degrade both predictive accuracy and the calibration of uncertainty estimates for deep learning models. This is worrying, because covariate shift is prevalent in a wide range of real world deployment settings. However, in this paper, we note that frequently there exists the potential to access small unlabeled batches of the shifted data just before prediction time. This interesting observation enables a simple but surprisingly effective method which we call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. Using this one line code change, we achieve state-of-the-art on recent covariate shift benchmarks and an mCE of 60.28% on the challenging ImageNet-C dataset; to our knowledge, this is the best result for any model that does not incorporate additional data augmentation or modification of the training pipeline. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness (e.g. deep ensembles) and combining the two further improves performance. Our findings are supported by detailed measurements of the effect of this strategy on model behavior across rigorous ablations on various dataset modalities. However, the method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift, and is therefore worthy of additional study. We include links to the data in our figures to improve reproducibility, including a Python notebooks that can be run to easily modify our analysis at this url.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift

Nado,

Padhy,

Sculley

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Several earlier comprehensive reviews on DA exist, each with a different focus, including visual applications (Csurka, 2017;Patel et al, 2015;Wilson and Cook, 2020), machine translation (MT) (Chu and Wang, 2018) pre-neural DA methods in NLP (Jiang, 2008;Margolis, 2011). Seminal surveys in machine learning on transfer learning include (Pan and Yang, 2009;Weiss et al, 2016;.…”

Section: Introductionmentioning

confidence: 99%

Neural Unsupervised Domain Adaptation in NLP---A Survey

Ramponi¹,

Plank²

2020

Preprint

View full text Add to dashboard Cite

Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks. In contrast, learning from unlabeled data, especially under domain shift, remains a challenge. Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data. This is a more challenging yet a more widely applicable setup. We outline methods, from early approaches in traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention. Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future intelligent NLP. 1

show abstract

“…One popular type of transfer learning is domain adaptation [122,118,166]. Domain adaptation is a type of transductive transfer learning, where the target task remains the same as the source, but the domain differs.…”

Section: Related Problemsmentioning

confidence: 99%

An Overview of Deep Semi-Supervised Learning

Ouali,

Hudelot,

Tami

2020

Preprint

View full text Add to dashboard Cite

Deep neural networks demonstrated their ability to provide remarkable performances on a wide range of supervised learning tasks (e.g., image classification) when trained on extensive collections of labeled data (e.g., ImageNet). However, creating such large datasets requires a considerable amount of resources, time, and effort. Such resources may not be available in many practical cases, limiting the adoption and the application of many deep learning methods. In a search for more data-efficient deep learning methods to overcome the need for large annotated datasets, there is a rising research interest in semi-supervised learning and its applications to deep neural networks to reduce the amount of labeled data required, by either developing novel methods or adopting existing semi-supervised learning frameworks for a deep learning setting. In this paper, we provide a comprehensive overview of deep semi-supervised learning, starting with an introduction to the field, followed by a summarization of the dominant semi-supervised approaches in deep learning 1 .

show abstract

A Survey of Unsupervised Deep Domain Adaptation

Cited by 13 publications

References 143 publications

Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift

Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift

Neural Unsupervised Domain Adaptation in NLP---A Survey

An Overview of Deep Semi-Supervised Learning

Contact Info

Product

Resources

About