Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Cha, Junbum; Lee, Kyung-Jae; Park, Sungrae; Chun, Sanghyuk

doi:10.48550/arxiv.2203.10789

Cited by 4 publications

(10 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the in-domain strategy by Gulrajani & Lopez-Paz (2020)) to select the best hyper-parameters, and report the average performance and standard deviation across 5 random seeds. Baselines : We compare our method against standard ERM training, which has proven to be a frustratingly difficult baseline (Gulrajani & Lopez-Paz, 2020), and also against several state of the art methods on this benchmark -SWAD (Cha et al, 2021), MIRO (Cha et al, 2022) and SMA (Arpit et al, 2021). Finally, we show that our approach can be effectively integrated with stochastic weight averaging to obtain further gains.…”

Section: Ood Generalization In a Real World Settingmentioning

confidence: 98%

Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks

Addepalli¹,

Nasery²,

Babu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep Neural Networks are known to be brittle to even minor distribution shifts compared to the training distribution. While one line of work has demonstrated that Simplicity Bias (SB) of DNNs -bias towards learning only the simplest features -is a key reason for this brittleness, another recent line of work has surprisingly found that diverse/ complex features are indeed learned by the backbone, and their brittleness is due to the linear classification head relying primarily on the simplest features. To bridge the gap between these two lines of work, we first hypothesize and verify that while SB may not altogether preclude learning complex features, it amplifies simpler features over complex ones. Namely, simple features are replicated several times in the learned representations while complex features might not be replicated. This phenomenon, we term Feature Replication Hypothesis, coupled with the Implicit Bias of SGD to converge to maximum margin solutions in the feature space, leads the models to rely mostly on the simple features for classification. To mitigate this bias, we propose Feature Reconstruction Regularizer (FRR) to ensure that the learned features can be reconstructed back from the logits. The use of FRR in linear layer training (FRR-L) encourages the use of more diverse features for classification. We further propose to finetune the full network by freezing the weights of the linear layer trained using FRR-L, to refine the learned features, making them more suitable for classification. Using this simple solution, we demonstrate up to 15% gains in OOD accuracy on the recently introduced semi-synthetic datasets with extreme distribution shifts. Moreover, we demonstrate noteworthy gains over existing SOTA methods on the standard OOD benchmark DomainBed as well.

show abstract

Section: Ood Generalization In a Real World Settingmentioning

confidence: 98%

Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks

Addepalli¹,

Nasery²,

Babu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We hypothesize that the gradient bias mentioned above could be relieved if the unobservable gradient g u minimizing risks in the unseen domains is computable. To achieve this, we borrow the assumption of Cha et al (2022) that large-scale pre-trained models are the approximation of the oracle model θ * which is optimally generalized for any domain D. Since the unobservable gradient g u cannot be computed from the unseen domains D u directly, we consider the direction from the current model θ to the oracle model θ * as the unobservable gradient g u . However, the oracle model is inaccessible in practice.…”

Section: Gestur: Grdient Estimation For Unseen Domain Risk Minimizati...mentioning

confidence: 99%

“…Implementation details. Our implementation is built on the codebase of Cha et al (2022). We use Adam optimizer (Kingma & Ba, 2015) for parameter optimization.…”

Section: Gestur: Grdient Estimation For Unseen Domain Risk Minimizati...mentioning

confidence: 99%

“…In every experiment, we search the optimal λ from {0.01, 0.05, 0.1, 0.5} and fix m as 0.999. Other hyperparameters such as learning rate, weight decay, and dropout are searched in the same way as Cha et al (2022). We explain more details of implementation in Appendix A.…”

Section: Gestur: Grdient Estimation For Unseen Domain Risk Minimizati...mentioning

confidence: 99%

“…Motivated by this, several studies have begun to leverage the generalization power of large-scale pre-trained models. Cha et al (2022) employ a pre-trained model for regularization, considering it as an approximation of the oracle model on any domain, and utilize a frozen Gradient "conflicts" (Yu et al, 2020;Mansilla et al, 2021) between g and g u (i.e., g • g u < 0) constantly occur throughout the whole fine-tuning iterations due to the gradient bias. Our proposed method reduces the number of gradient conflicts by adding the estimated unobservable gradient gu to the biased gradient g. This observation indicates that the gradient bias is relieved with the estimated gradient during model optimization.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models

Lew¹,

Son²,

Chang³

2023

Preprint

View full text Add to dashboard Cite

Domain generalization aims to build generalized models that perform well on unseen domains when only source domains are available for model optimization. Recent studies have demonstrated that large-scale pre-trained models could play an important role in domain generalization by providing their generalization power. However, large-scale pre-trained models are not fully equipped with target task-specific knowledge due to a discrepancy between the pre-training objective and the target task. Although the task-specific knowledge could be learned from source domains by fine-tuning, this hurts the generalization power of the pretrained models because of gradient bias toward the source domains. To address this issue, we propose a new domain generalization method that estimates unobservable gradients that reduce potential risks in unseen domains, using a largescale pre-trained model. Our proposed method allows the pre-trained model to learn task-specific knowledge further while preserving its generalization ability with the estimated gradients. Experimental results show that our proposed method outperforms baseline methods on DOMAINBED, a standard benchmark in domain generalization. We also provide extensive analyses to demonstrate that the estimated unobserved gradients relieve the gradient bias, and the pre-trained model learns the task-specific knowledge without sacrificing its generalization power.

show abstract

DGM-DR: Domain Generalization with Mutual Information Regularized Diabetic Retinopathy Classification

Matsun,

Mohamed,

Chokuwa

et al. 2023

Domain Adaptation and Representation Transfer

View full text Add to dashboard Cite

Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Cited by 4 publications

References 25 publications

Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks

Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks

Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models

DGM-DR: Domain Generalization with Mutual Information Regularized Diabetic Retinopathy Classification

Contact Info

Product

Resources

About