Combining Diverse Feature Priors

Abstract

To improve model generalization, model designers often restrict the features that their models use, either implicitly or explicitly. In this work, we explore the design space of leveraging such feature priors by viewing them as distinct perspectives on the data. Specifically, we find that models trained with diverse sets of feature priors have less overlapping failure modes, and can thus be combined more effectively. Moreover, we demonstrate that jointly training such models on additional (unlabeled) data allows them to correct each other's mistakes, which, in turn, leads to better generalization and resilience to spurious correlations. 1

InBiaseD: Inductive Bias Distillation to Improve Generalization and Robustness through Shape-awareness

“…Ensemble approaches: To obtain better generalization across domains, Mancini et al (2018) built an ensemble of multiple domain-specific classifiers, each attributing to a different cue. Jain et al (2021) trained multiple networks separately, each with a different kind of bias. The ensemble of these biased networks was used to produce pseudo labels for unlabeled data.…”

“…We investigate this effect on three different datasets. We create a synthetic dataset, Tinted-STL-10, by adding a class-specific tint to the original STL-10 data following Jain et al (2021). This tint is only added to the training set (see samples in Figure 9) and not to the test set.…”

“…For the third dataset, we consider the gender classification task based on CelebA (Liu et al, 2015). We create a skewed-CelebA dataset that consists of only "blond-females" (B-F) and "non-blond-males" (NB-M) samples and use this as the training set following Jain et al (2021). The test set contains all four combinations of hair color and gender.…”