A systematic study of the class imbalance problem in convolutional neural networks

Abstract

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

Identifying animal species in camera trap images using deep learning and citizen science

“…Norouzzadeh et al. () addressed this problem by placing higher weights on rare classes during model training but were not able to systematically improve accuracies for rare species (see Buda, Maki, and Mazurowski () for strategies to address class imbalance in modelling).…”

A convolutional neural network to filter artifacts in spectroscopic MRI

“…This suggests that the CNN is robust to the class imbalance of the ground‐truth data set, but a more balanced data set could potentially improve outcomes. Of note, the imbalance is relatively small, being only a factor of approximately 2.6, whereas in other domains in which deep learning is been applied the imbalance can be several orders of magnitude …”

An Improved 3D Shape Recognition Method Based on Panoramic View

“…The balanced training set did not get too much better result than the original training set. It is well known that training a network with an unbalanced dataset tends to harm those classes with the least number of examples and benefit those with the most [24], and it is not clear how the imbalanced attribute affects the experimental results. Sometimes balancing the training set improves the accuracy of the classes with fewer examples but harms the success rate for classes with more samples [7].…”