Do Deep Networks Transfer Invariances Across Classes?

Abstract

To generalize well, classifiers must learn to be invariant to nuisance transformations that do not alter an input's class. Many problems have "class-agnostic" nuisance transformations that apply similarly to all classes, such as lighting and background changes for image classification. Neural networks can learn these invariances given sufficient data, but many real-world datasets are heavily class imbalanced and contain only a few examples for most of the classes. We therefore pose the question: how well do neural networks transfer class-agnostic invariances learned from the large classes to the small ones? Through careful experimentation, we observe that invariance to class-agnostic transformations is still heavily dependent on class size, with the networks being much less invariant on smaller classes. This result holds even when using data balancing techniques, and suggests poor invariance transfer across classes. Our results provide one explanation for why classifiers generalize poorly on unbalanced and long-tailed distributions. Based on this analysis, we show how a generative approach for learning the nuisance transformations can help transfer invariances across classes and improve performance on a set of imbalanced image classification benchmarks. Source code for our experiments is available at https://github.com/AllanYangZhou/ generative-invariance-transfer.

Discrete Key-Value Bottleneck

“…We now consider the setting where we are confronted with training for five additional epochs under an imbalanced dataset shift. In particular, we construct a strongly imbalanced dataset shift, with an exponential long-tailed frequency distribution of the classes with an imbalance ratio of 100 as in [68]. Our result is summarized in Figure 7.…”

C-Mixup: Improving Generalization in Regression

“…Data Augmentation and Mixup. Various data augmentation strategies have been proposed to improve the generalization of deep neural networks, including directly augmenting images with manually designed strategies (e.g., whitening, cropping) [37], generating more examples with generative models [3,7,88,85], and automatically finding augmentation strategies [10,11,43]. Mixup [82] and its variants [9, 12, 20, 24, 26, 32, 33, 44, 45, 52, 66-68, 81, 87] propose to improve generalization by linear interpolating input features of a pair of examples and their corresponding labels.…”