A Fine-Grained Analysis on Distribution Shift

Wiles, Olivia; Gowal, Sven; Stimberg, Florian; Alvise-Rebuffi, Sylvestre; Ira, Ktena,; Dvijotham, Krishnamurthy; Cemgil, Taylan

doi:10.48550/arxiv.2110.11328

Cited by 12 publications

(17 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparing Datasets and Subsets Thereof Such comparisons have various applications, e.g., to assessing domain suitability in transfer learning and analyzing distribution shift [57]. We presented two simple examples of comparing specific features between two image datasets (Figure 1) or between two splits of CityScapes (Figure 6).…”

Section: Visually Linking Model Results With Dataset Propertiesmentioning

confidence: 99%

A Tour of Visualization Techniques for Computer Vision Datasets

Alsallakh¹,

Bhattacharya²,

Feng³

et al. 2022

Preprint

View full text Add to dashboard Cite

We survey a number of data visualization techniques for analyzing Computer Vision (CV) datasets. These techniques help us understand properties and latent patterns in such data, by applying dataset-level analysis. We present various examples of how such analysis helps predict the potential impact of the dataset properties on CV models and informs appropriate mitigation of their shortcomings. Finally, we explore avenues for further visualization techniques of different modalities of CV datasets as well as ones that are tailored to support specific CV tasks and analysis needs.

show abstract

Section: Visually Linking Model Results With Dataset Propertiesmentioning

confidence: 99%

A Tour of Visualization Techniques for Computer Vision Datasets

Alsallakh¹,

Bhattacharya²,

Feng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides, there are two popular approaches, domain-invariant representation learning and invariant risk minimization, which we will discuss in detail below. In addition to algorithms, there are works that propose theoretical frameworks for DG (Zhang et al, 2021;Ye et al, 2021), or empirically examine DG al-gorithms over various benchmarks (Gulrajani & Lopez-Paz, 2021;Koh et al, 2021;Wiles et al, 2021). Domain-Invariant Representation Learning.…”

Section: Related Workmentioning

confidence: 99%

Provable Domain Generalization via Invariant-Feature Subspace Recovery

Wang¹,

Si²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Domain generalization asks for models trained on a set of training environments to perform well on unseen test environments. Recently, a series of algorithms such as Invariant Risk Minimization (IRM) has been proposed for domain generalization. However, Rosenfeld et al. (2021) shows that in a simple linear data model, even if non-convexity issues are ignored, IRM and its extensions cannot generalize to unseen environments with less than d s `1 training environments, where d s is the dimension of the spuriousfeature subspace. In this paper, we propose to achieve domain generalization with Invariantfeature Subspace Recovery (ISR). Our first algorithm, ISR-Mean, can identify the subspace spanned by invariant features from the first-order moments of the class-conditional distributions, and achieve provable domain generalization with d s `1 training environments under the data model of Rosenfeld et al. (2021). Our second algorithm, ISR-Cov, further reduces the required number of training environments to Op1q using the information of second-order moments. Notably, unlike IRM, our algorithms bypass non-convexity issues and enjoy global convergence guarantees. Empirically, our ISRs can obtain superior performance compared with IRM on synthetic benchmarks. In addition, on three real-world image and text datasets, we show that ISR-Mean can be used as a simple yet effective post-processing method to increase the worst-case accuracy of trained models against spurious correlations and group shifts. The code is released at https: //github.com/Haoxiang-Wang/ISR.

show abstract

“…History-restricted algorithms are a natural approach to online learning over non-stationary time series, which is a common problem in practical industry settings [Huyen, 2022] and which has been studied for many years [Sugiyama and Kawanabe, 2012, Wiles et al, 2021, Wu, 2021, Rabanser et al, 2019. In particular, one can view history-restricted online learners as adapting to non-stationary structure in the data -thus, one may not need to go through the whole process of detecting a distribution shift and then deciding to re-train -ideally the learning algorithm is adaptive and automatically takes such eventualities into account.…”

Section: A3 Shifting Data Distributionsmentioning

confidence: 99%

History-Restricted Online Learning

Schneider¹,

Vodrahalli²

2022

Preprint

View full text Add to dashboard Cite

We introduce the concept of history-restricted no-regret online learning algorithms. An online learning algorithm A is M -history-restricted if its output at time t can be written as a function of the M previous rewards. This class of online learning algorithms is quite natural to consider from many perspectives: they may be better models of human agents and they do not store long-term information (thereby ensuring "the right to be forgotten"). We first demonstrate that a natural approach to constructing history-restricted algorithms from mean-based no-regret learning algorithms (e.g. running Hedge over the last M rounds) fails, and that such algorithms incur linear regret. We then construct a history-restricted algorithm that achieves a per-round regret of Θ(1/ √ M ), which we complement with a tight lower bound. Finally, we empirically explore distributions where history-restricted online learners have favorable performance compared to other no-regret algorithms.

show abstract

A Fine-Grained Analysis on Distribution Shift

Cited by 12 publications

References 28 publications

A Tour of Visualization Techniques for Computer Vision Datasets

A Tour of Visualization Techniques for Computer Vision Datasets

Provable Domain Generalization via Invariant-Feature Subspace Recovery

History-Restricted Online Learning

Contact Info

Product

Resources

About