SetConv: A New Approach for Learning from Imbalanced Data

Abstract

For many real-world classification problems, e.g., sentiment classification, most existing machine learning methods are biased towards the majority class when the Imbalance Ratio (IR) is high. To address this problem, we propose a set convolution (SetConv) operation and an episodic training strategy to extract a single representative for each class, so that classifiers can later be trained on a balanced class distribution. We prove that our proposed algorithm is permutation-invariant despite the order of inputs, and experiments on multiple large-scale benchmark text datasets show the superiority of our proposed framework when compared to other SOTA methods.

Error-Sensitive Evaluation for Ordinal Target Variables

“…2) MC over-predicts majority classes in both datasets (2s and 3s for MR and 1s and 10s for IMDb) while under-predicting the others (except 2s and 3s in IMDb). These results are in line with the common observation that MC models tend to overfit on the majority classes in im- balanced datasets, which motivates the use of "oversampling" or class balancing (Buda et al, 2018;Chawla et al, 2002;Tepper et al, 2020;Gao et al, 2020). OR, in contrast, provides a better fit for MR (slightly under-predicting for 1s), but significantly under-predicts on IMDb majority classes, displaying a much flatter distribution of predictions.…”

A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing

“…More concrete definitions, e.g., regarding the relative share up to which a class is seen as a minority class, depend on the task, dataset and labelset size. Much research focuses on improving all minority classes equally while maintaining or at least monitoring majority class performance (e.g., Huang et al, 2021;Yang et al, 2020;Spangher et al, 2021). We next discuss prototypical types of imbalance (Sec.…”

“…Under imbalance, two issues arise. First, although class-specific weights have been used with BCE (e.g., Yang et al, 2020), their effect on minority classes is less clear than in the single-label case. For each instance, all classes contribute to BCE, with the labels not assigned to the instance (called negative classes) included via (1−y j ) log(1−p j ).…”

“…SetConv (Gao et al, 2020) and ProtoBERT (Tänzer et al, 2022) learn representatives for each class using support sets and classify an input (the query) based on its similarity to these representatives. SetConv applies convolution kernels that capture intra-and inter-class correlations to extract class representatives.…”

“…Which categories matter is highly task-specific and may even depend on the intended downstream use. Developing methods that improve model performance in imbalanced data settings has been an active area for decades (e.g., Bruzzone and Serpico, 1997;Japkowicz et al, 2000;Estabrooks and Japkowicz, 2001;Park and Zhang, 2002;Tan, 2005), and is recently gaining momentum in the context of maturing neural approaches (e.g., Buda et al, 2018;Kang et al, 2020;Yang et al, 2020;Jiang et al, 2021;Spangher et al, 2021). The problem is exacerbated when classes overlap in the feature space (Lin et al, 2019;Tian et al, 2020).…”