Out-of-Distribution Generalization via Risk Extrapolation (REx)

Abstract

Generalizing outside of the training distribution is an open challenge for current machine learning systems. A weak form of out-of-distribution (OoD) generalization is the ability to successfully interpolate between multiple observed distributions. One way to achieve this is through robust optimization, which seeks to minimize the worstcase risk over convex combinations of the training distributions. However, a much stronger form of OoD generalization is the ability of models to extrapolate beyond the distributions observed during training. In pursuit of strong OoD generalization, we introduce the principle of Risk Extrapolation (REx). REx can be viewed as encouraging robustness over affine combinations of training risks, by encouraging strict equality between training risks. We show conceptually how this principle enables extrapolation, and demonstrate the effectiveness and scalability of instantiations of REx on various OoD generalization tasks. Our code can be found at https://github.com/capybaralet/ REx_code_release.

On the Impact of Spurious Correlation for Out-of-distribution Detection

“…A plethora of algorithms are proposed: learning invariant representation across domains [7,21,38,20], minimizing the weighted combination of risks from training domains [35], using different risk penalty terms to facilitate invariance prediction [1,17], causal inference approaches [31], and forcing the learned representation different from a set of pre-defined biased representations [2], mixup-based approaches [48,41,26], etc. A recent study [10] shows that no domain generalization methods achieve superior performance than ERM across a broad range of datasets.…”

“…Extension: Empirical Validation of Theoretical Analysis. To further validate our analysis above, we comprehensively evaluate the OOD detection performance of models that are trained with recent prominent domain invariance learning objectives [1,2,17,7,21,35] (Section E in Appendix). The results align with our theoretical analysis.…”

“…Follow-up works proposed several variations, based on different notions of invariance. In particular, [17] proposed Risk Extrapolation (REx), which aims to achieve stronger invariance p(y|Φ(x)) by penalizing the variance of risks of environments. Other approaches have proposed to remove the predictability of p(e|Φ(x)) through domain adversarial losses such as DANN [7] and CDANN [21] (adapted for domain generalization).…”

“…In-distribution Classification Performance. 99.98 ± 0.02 100.00 ± 0.00 99.99 ± 0.02 GDRO [35] 99.97 ± 0.04 99.98 ± 0.03 99.98 ± 0.02 REx [17] 100.00 ± 0.00 99.99 ± 0.02 99.99 ± 0.02 DANN [7] 99.97 ± 0.02 99.99 ± 0.02 99.99 ± 0.02 CDANN [21] 99.97 ± 0.02 99.99 ± 0.02 99.98 ± 0.02 95.97 ± 0.62 GDRO [35] 95.74 ± 0.54 REx [17] 95.49 ± 0.77 DANN [7] 96.27 ± 0.25 CDANN [21] 94.74 ± 0.63…”

Class-conditioned Domain Generalization via Wasserstein Distributional Robust Optimization

“…In an ideal case, even simple k-NN based method can perform well. However, when the variation among class-conditionals of the same class is large, i.e., the closest conditional distribution to T (X|Y = y) is some S(X|Y = y ) of class y = y (Figure 1 Right), aforementioned methods may not perform well (Krueger et al, 2020).…”