Stop Oversampling for Class Imbalance Learning: A Review

Abstract

For the last two decades, oversampling has been employed to overcome the challenge of learning from imbalanced datasets. Many approaches to solving this challenge have been offered in the literature. Oversampling, on the other hand, is a concern. That is, models trained on fictitious data may fail spectacularly when put to real-world problems. The fundamental difficulty with oversampling approaches is that, given a real-life population, the synthesized samples may not truly belong to the minority class. As a result, training a classifier on these samples while pretending they represent minority may result in incorrect predictions when the model is used in the real world. We analyzed a large number of oversampling methods in this paper and devised a new oversampling evaluation system based on hiding a number of majority examples and comparing them to those generated by the oversampling process. Based on our evaluation system, we ranked all these methods based on their incorrectly generated examples for comparison. Our experiments using more than 70 oversampling methods and nine imbalanced real-world datasets reveal that all oversampling methods studied generate minority samples that are most likely to be majority. Given data and methods in hand, we argue that oversampling in its current forms and methodologies is unreliable for learning from class imbalanced data and should be avoided in real-world applications.

Are Classifiers Trained on Synthetic Data Reliable? An XAI Study

“…The number of nearest neighbors is a hyper-parameter for SMOTE and is usually selected based on performance over a specific metric, e.g., model accuracy on the validation set. SMOTE, however, does not consider the underlying class distribution and is prune to over-fitting, class overlap, and noisy sample generation [14]. Although a number of potential improvements over SMOTE have been proposed, (see [14] and references therein), over time, SMOTE has become more popular among researchers and has been seen as the default upsampling method in the field of imbalanced learning.…”

“…SMOTE, however, does not consider the underlying class distribution and is prune to over-fitting, class overlap, and noisy sample generation [14]. Although a number of potential improvements over SMOTE have been proposed, (see [14] and references therein), over time, SMOTE has become more popular among researchers and has been seen as the default upsampling method in the field of imbalanced learning. Thus, we chose to include SMOTE as a benchmark method in our analysis.…”

“…Thus these methods might over-fit to the available minority samples. Consequently, the models trained on synthetically generated data may fail all-together when facing the real-world problem [14]. Another major problem with the synthetically generated samples is that they may belong to a different class in the real world, even if they are similar to the minority samples.…”

“…Recently, authors in [14] have compared the performance of 72 traditional upsampling methods on 9 real-world imbalanced datasets and have shown that a high number of synthetic minority samples do, in fact, belong to the majority class. The authors argued against using synthetic samples for model training altogether.…”

Are Classifiers Trained on Synthetic Data Reliable? An XAI Study

“…There are a number of AE variants in literature [14], out of which the Variational Autoencoders (VAEs) [12] are the most popular. In vanilla AE, the encoder outputs a single latent vector z directly, however, in VAEs, the encoder outputs two vectors; a mean vector µ and a variance vector σ.…”