Reducing Domain Gap by Reducing Style Bias

Nam, Hyeonseob; Lee, HyunJae; Park, Jongchan; Yoon, Wonjun; Yoo, Donggeun

doi:10.1109/cvpr46437.2021.00858

Cited by 197 publications

(104 citation statements)

References 24 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…Compared with the state-of-the-art methods, MIRO achieves the best performances in all benchmarks, except PACS. Especially, MIRO remarkably outperforms previous state-of-the-arts: +1.3pp in OfficeHome (mDSDI [59]; 69.2% → 70.5%) and +1.8pp in TerraIncognita (SagNet [29]; 48.6% → 50.4%). Considering the experiment setup with 5 datasets and 22 target domains, the experiment results demonstrate the effectiveness of MIRO to the diverse visual data types.…”

Section: Resultsmentioning

confidence: 89%

“…The trained model on multiple source domains is evaluated on an unseen domain (e.g., art painting) to measure the robustness against distribution shifts. The existing DG approaches have tried to learn invariant features across multiple domains by minimizing feature divergences between the source domains [10][11][12][13][14][15][16], normalizing domain-specific gradients based on meta-learning [17][18][19][20][21], robust optimization [22][23][24][25], or augmenting source domain examples [26][27][28][29][30][31][32]. However, recent studies [33,34] have shown that simple baselines without learning invariant features are comparable or even outperform the existing DG methods on the diverse DG benchmarks with a fair hyperparameter selection protocol when a model becomes larger (e.g., from ResNet-18 to ResNet-50 [35]).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Cha¹,

Lee²,

Park³

et al. 2022

Preprint

View full text Add to dashboard Cite

Domain generalization (DG) aims to learn a generalized model to an unseen target domain using only limited source domains. Previous attempts to DG fail to learn domain-invariant representations only from the source domains due to the significant domain shifts between training and test domains. Instead, we reformulate the DG objective using mutual information with the oracle model, a model generalized to any possible domain. We derive a tractable variational lower bound via approximating the oracle model by a pre-trained model, called Mutual Information Regularization with Oracle (MIRO). Our extensive experiments show that MIRO significantly improves the out-of-distribution performance. Furthermore, our scaling experiments show that the larger the scale of the pre-trained model, the greater the performance improvement of MIRO. Source code is available at https://github.com/kakaobrain/miro.

show abstract

Section: Resultsmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Cha¹,

Lee²,

Park³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The main idea in DB is to utilize known biases (or identify unknown biases) in the data distribution, model these biases in the training pipeline, and use this knowledge to train robust classifiers (Clark et al, 2019;Bhargava et al, 2021). In the image classification literature, there is growing consensus on enforcing a consistency on different views (or augmentations) of an image in order to achieve debiasing (Hendrycks et al, 2020c;Xu et al, 2020;Chai et al, 2021;Nam et al, 2021). Unlike DF, model de-biasing does not directly alter the training distribution, but instead allows the model to learn which biases to ignore.…”

Section: Categorization Of Domain Generalization Methodsmentioning

confidence: 99%

Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness

Gokhale¹,

Mishra²,

Luo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data modification, either via additional training datasets, data augmentation, debiasing, and dataset filtering, has been proposed as an effective solution for generalizing to outof-domain (OOD) inputs, in both natural language processing and computer vision literature. However, the effect of data modification on adversarial robustness remains unclear. In this work, we conduct a comprehensive study of common data modification strategies and evaluate not only their in-domain and OOD performance, but also their adversarial robustness (AR). We also present results on a twodimensional synthetic dataset to visualize the effect of each method on the training distribution. This work serves as an empirical study towards understanding the relationship between generalizing to unseen domains and defending against adversarial perturbations. Our findings suggest that more data (either via additional datasets or data augmentation) benefits both OOD accuracy and AR. However, data filtering (previously shown to improve OOD accuracy on natural language inference) hurts OOD accuracy on other tasks such as question answering and image classification. We provide insights from our experiments to inform future work in this direction.

show abstract

“…• SagNet: Style Agnostic Networks by Nam et al [36] • ARM: Adaptive Risk Minimization by Zhang et al [37] • VREx: Variance Risk Extrapolation by Krueger et al [15] • RSC: Representation Self-Challenging by Huang et al [38] • SD: Spectral Decoupling by Pezeshki et al [39] • AND-mask: Learning Explanations that are Hard to Vary by Parascandolo et al [12] Employed Architecture In Table 5, we detail the architecture used for experimentation. For the MLP architecture, it's depth and width are defined as hyperparameter included in the hyperparameter search.…”

Section: Conditionmentioning

confidence: 99%

SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization

Shahtalebi,

Gagnon-Audet,

Laleh

et al. 2021

Preprint

View full text Add to dashboard Cite

A major bottleneck in the real-world applications of machine learning models is their failure in generalizing to unseen domains whose data distribution is not i.i.d to the training domains. This failure often stems from learning non-generalizable features in the training domains that are spuriously correlated with the label of data. To address this shortcoming, there has been a growing surge of interest in learning good explanations that are hard to vary, which is studied under the notion of Out-of-Distribution (OOD) Generalization. The search for good explanations that are invariant across different domains can be seen as finding local (global) minimas in the loss landscape that hold true across all of the training domains. In this paper, we propose a masking strategy, which determines a continuous weight based on the agreement of gradients that flow in each edge of network, in order to control the amount of update received by the edge in each step of optimization. Particularly, our proposed technique referred to as "Smoothed-AND (SAND)masking", not only validates the agreement in the direction of gradients but also promotes the agreement among their magnitudes to further ensure the discovery of invariances across training domains. SAND-mask is validated over the Domainbed benchmark for domain generalization and significantly improves the state-of-the-art accuracy on the Colored MNIST dataset, while providing competitive results on other domain generalization datasets.Preprint. Under review.

show abstract

Reducing Domain Gap by Reducing Style Bias

Cited by 197 publications

References 24 publications

Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness

SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization

Contact Info

Product

Resources

About