Assaying Out-Of-Distribution Generalization in Transfer Learning

Abstract

Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many-and few-shot setting. Our findings confirm that in-and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.

Empirical Study on Optimizer Selection for Out-of-Distribution Generalization

“…These results confirm that diminishing returns, in particular, are not necessarily linear even before probit transformation. This is consistent with the recent point by Wenzel et al (2022); Teney et al (2022) that the accuracy of IID and OOD differ depending on the dataset. Furthermore, as recent work, Baek et al (2022) states, whether or not the linear return actually occurs depends on the problem set.…”

The transformative potential of artificial intelligence in solid organ transplantation

“…So far, there is no consensus on how to address this issue. Nevertheless, promising work in causal representation learning (59) is encouraging and raises hope that robustness will soon be achieved by AI models.…”