Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

Abstract

Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains 1 .

Adversarial Examples for Good: Adversarial Examples Guided Imbalanced Learning

“…FashionMNIST [], CIFAR-10 and CIFAR-100 [] with controllable degrees of data imbalance. Follow the setting in [3], the magnitude of the long-tailed imbalance declines exponentially with class size. The imbalance ratio ρ represents the ratio between the samples of the most and least frequent classes.…”

“…[9,1,2]. It is usually the case that real-world data exhibit a imbalanced distribution, and highly skewed data can adversely affect the effectiveness of machine learning [10,3]. Re-sampling [11] and re-weighting [12,10,13] are traditional methods towards addressing imbalanced data.…”

“…Moreover, two resampling methods, under-sampling frequent classes and oversampling minority classes, have been extensively discussed in previous studies. Also, we saw success with new perspectives, such as deferred re-balancing [3] schedule and decoupled training [14]. In most prior studies of imbalanced data, the emphasis has been placed on long-tailed distributions.…”

“…As a result, deep learning algorithms usually perform poorly or even collapse when the training data suffers from heavy class-imbalance, especially for highly skewed data [2]. Due to the imbalanced data distribution, neural networks can over-fitting to the minority classes, which leads to the deviation of the learned classification boundary [3] .…”

“…Recent research aims to modify the data sampler to balance class frequency during optimization (re-sampling), and modify the weights of the classification loss to increase the importance of tail classes (re-weighting). In general, reweighting and re-sampling are two main approaches that are commonly used to learn long-tailed data [3]. In contrast to previous works, we provide a new perspective on how to deal with imbalanced data: adjust the biased decision boundary by training with Guiding Adversarial Examples (GAEs).…”