Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency

Abstract

Visual domain adaptation involves learning to classify images from a target visual domain using labels available in a different source domain. A range of prior work uses adversarial domain alignment to try and learn a domain invariant feature space, where a good source classifier can perform well on target data. This however, can lead to errors where class A features in the target domain get aligned to class B features in source. We show that in the presence of a few target labels, simple techniques like selfsupervision (via rotation prediction) and consistency regularization can be effective without any adversarial alignment to learn a good target classifier. Our Pretraining and Consistency (PAC) approach, can achieve state of the art accuracy on this semi-supervised domain adaptation task, surpassing multiple adversarial domain alignment methods, across multiple datasets. Notably, it outperforms all recent approaches by 3-5% on the large and challenging DomainNet benchmark, showing the strength of these simple techniques in fixing errors made by adversarial alignment.

Extending the WILDS Benchmark for Unsupervised Adaptation

“…Self-supervised methods learn useful representations by training on unlabeled data via auxiliary proxy tasks. Common approaches include reconstruction tasks (Vincent et al, 2008;Erhan et al, 2010;Devlin et al, 2019;Gidaris et al, 2018;Lewis et al, 2020), and contrastive learning (He et al, 2020;Chen et al, 2020b;Caron et al, 2020;Radford et al, 2021b), and recent work has shown that self-supervised methods can reduce dependence on spurious correlations and improve performance on domain adaptation tasks (Wang et al, 2021;Tsai et al, 2021;Mishra et al, 2021). We use these self-supervision methods for unsupervised adaptation by first pre-training models on the unlabeled data, and then finetuning them on the labeled source data (Shen et al, 2021).…”