Deep Generative Models to Counter Class Imbalance: A Model-Metric Mapping With Proportion Calibration Methodology

Abstract

The most pervasive segment of techniques in managing class imbalance in machine learning are re-sampling-based methods. The emergence of deep generative models for augmenting the size of the under-represented class, prompts one to review the question of the suitability of the model chosen for data augmentation with the metric selected for the-goodness-of classification. This work defines this suitability by using newly-sampled data points from each generative model first to the degree of parity, and studying classification performance on a large set of metrics. We extend the investigation to different proportions of augmented data points for identifying the sensitivity of the metric to the degree of imbalance, leading to the discovery of an optimum proportion against the metric. The models used are GAN, VAE and RBM and the metrics include Precision, Recall, F1-Score, AUC, G-Mean and Balanced Accuracy. We offer a comparison of these models with the established class of data synthesizing counterparts on the aforementioned metrics. Deep generative models outperform the state-of-the-art on 5 metrics on multiple datasets and also comprehensively surpass the baselines. This work thereby recommends the following model-metric mappings: VAE for high Precision and F1-Score, RBM for high Recall and GAN for high AUC, G-Mean and Balanced Accuracy under various recommended proportions of the minority class.

Learning From Few Cyber-Attacks: Addressing the Class Imbalance Problem in Machine Learning-Based Intrusion Detection in Software-Defined Networking

“…The core idea behind generative modeling in the context of tackling the class imbalance problem is to estimate the probability density function describing the data and generate new data instances in a random fashion [39] in order to balance the data distribution in an otherwise imbalanced dataset. Generative models typically construct a latent space that aims to capture the direct cause of the target variable.…”

The use of generative adversarial networks to alleviate class imbalance in tabular data: a survey

“…Where most research has focused on modifying GAN architecture to achieve optimal results in class imbalanced settings, Mizra et al [79] posed a distinct yet equally important question: given optimized performance on a desired evaluation metric, what data augmentation method and proportion of synthetic sample injection should be used? The resulting framework, termed Model-Metric Mapper methodology, or MMM, can conversely offer a procedural and hierarchical approach to guide the practitioner toward proper model selection based on desired evaluation metric.…”

“…Borderline-SMOTE and ADAYSN are both frequently cited as baseline methods, and consequently both [20] and [14] possess a high in-degree count. The use of conditional GANs for data generation based on class labels is addressed in [52,79], and [84] (all with five or six in-degrees). With five in-network citations, [85] offers advice vis-à-vis best practices for hyper-tuning GANs at time of publication, though this work is mainly done with computer vision tasks in mind.…”

Ensemble Partition Sampling (EPS) for Improved Multi-Class Classification

“…The author identifies seven vital areas of research in this topic, covering the full spectrum of learning from imbalanced data such as classification, regression, clustering, data streams, big data analytics and applications. Fanny et al [11], Ming et al [12], Zhai et al [13], and Mirza et al [14] propose different deep learning approaches to address class imbalance, Fanny et al [11] proposed a method based on Class Expert Generative Adversarial Network (CE-GAN). In this approach, a GAN is trained for each minority class, with the generator network being conditioned on the class label.…”

“…This approach improves the performance of minority classes by increasing the diversity of the training data. Mirza et al [14] proposed deep generative models to counter class imbalance. They proposed two approaches: the first approach involves using a variational autoencoder to generate synthetic samples for the minority class, while the second approach involves using a generative adversarial network to generate synthetic samples.…”