Provable Domain Generalization via Invariant-Feature Subspace Recovery

Abstract

Domain generalization asks for models trained on a set of training environments to perform well on unseen test environments. Recently, a series of algorithms such as Invariant Risk Minimization (IRM) has been proposed for domain generalization. However, Rosenfeld et al. (2021) shows that in a simple linear data model, even if non-convexity issues are ignored, IRM and its extensions cannot generalize to unseen environments with less than d s `1 training environments, where d s is the dimension of the spuriousfeature subspace. In this paper, we propose to achieve domain generalization with Invariantfeature Subspace Recovery (ISR). Our first algorithm, ISR-Mean, can identify the subspace spanned by invariant features from the first-order moments of the class-conditional distributions, and achieve provable domain generalization with d s `1 training environments under the data model of Rosenfeld et al. (2021). Our second algorithm, ISR-Cov, further reduces the required number of training environments to Op1q using the information of second-order moments. Notably, unlike IRM, our algorithms bypass non-convexity issues and enjoy global convergence guarantees. Empirically, our ISRs can obtain superior performance compared with IRM on synthetic benchmarks. In addition, on three real-world image and text datasets, we show that ISR-Mean can be used as a simple yet effective post-processing method to increase the worst-case accuracy of trained models against spurious correlations and group shifts. The code is released at https: //github.com/Haoxiang-Wang/ISR.

A Review of the Role of Causality in Developing Trustworthy AI Systems

“…Pre-Processing Data Augmentation [106,121,144,158] Problem Abstraction [107,281] In-Processing Objective (Loss) Design [13,63,150,169] [ 258,280,307] Architecture Design [85,146,298] Post-Processing Altering Predictions [45,279] Model Selection [131] Fig. 5.…”

“…Using such a counterfactual regularization, Chen et al [45] successfully improved the quality of trajectory predictions in multi-domain settings. Wang et al [279] introduce an alternative approach called Invariant-Feature Subspace Recovery (ISR). The authors first extract the feature representation of the given data via the model's hidden layers.…”

“…− → Used by:[279] • CelebA[149]: A face image dataset containing 202,599 images of size 178×218 from 10,177 unique celebrities. Each image is annotated with 40 binary facial attributes (e.g., Is this person smiling?)…”

“…and five landmark positions describing the 2D position of the eyes, the nose, and the mouth (split into left and right side of the mouth). − → Used by:[279] • a variety of adversarial attacks for image classification through AutoAttack. It also provides multiple continuously updated leaderboards of the most robust models, which allows for direct comparisons between causal and non-causal methods.…”