Re-embedding Difficult Samples via Mutual Information Constrained Semantically Oversampling for Imbalanced Text Classification

Tian, Jiachen; Chen, Shizhan; Zhang, Xiaowang; Xiong, Deyi; Wu, Shaojuan; Dou, Chunliu

doi:10.18653/v1/2021.emnlp-main.252

Cited by 7 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…h . This goal can be finished with the Mutual Information Neural Estimator (MINE) (Belghazi et al 2018;Hjelm et al 2019;Tian et al 2021;Mroueh et al 2021). Specifically, we use a discriminator T ω a h with parameters ω a h to maximize the mutual information between V a h and V + h , and the loss function can be defined as follows:…”

Section: Gumbel-attack Expertmentioning

confidence: 99%

Reducing Sentiment Bias in Pre-trained Sentiment Classification via Adaptive Gumbel Attack

Tian,

Chen,

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have recently enabled rapid progress on sentiment classification under the pre-train and fine-tune paradigm, where the fine-tuning phase aims to transfer the factual knowledge learned by PLMs to sentiment classification. However, current fine-tuning methods ignore the risk that PLMs cause the problem of sentiment bias, that is, PLMs tend to inject positive or negative sentiment from the contextual information of certain entities (or aspects) into their word embeddings, leading them to establish spurious correlations with labels. In this paper, we propose an adaptive Gumbel-attacked classifier that immunes sentiment bias from an adversarial-attack perspective. Due to the complexity and diversity of sentiment bias, we construct multiple Gumbel-attack expert networks to generate various noises from mixed Gumbel distribution constrained by mutual information minimization, and design an adaptive training framework to synthesize complex noise by confidence-guided controlling the number of expert networks. Finally, we capture these noises that effectively simulate sentiment bias based on the feedback of the classifier, and then propose a multi-channel parameter updating algorithm to strengthen the classifier to recognize these noises by fusing the parameters between the classifier and each expert network. Experimental results illustrate that our method significantly reduced sentiment bias and improved the performance of sentiment classification.

show abstract

Section: Gumbel-attack Expertmentioning

confidence: 99%

Reducing Sentiment Bias in Pre-trained Sentiment Classification via Adaptive Gumbel Attack

Tian,

Chen,

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…However, evaluating on naturally imbalanced data provides evidence of a method's real-world effectiveness. Some recent studies combine both types of evaluation (e.g., Tian et al, 2021;Subramanian et al, 2021;Jang et al, 2021). Many NLP tasks require treating a large, often heterogenous catch-all class that contains all instances that are not of interest to the task, while the remaining (minority) classes are approximately same-sized.…”

Section: Controlled Vs Real-world Class Imbalancementioning

confidence: 99%

“…MISO (Tian et al, 2021) generates new instances by transforming the representations of minority class instances that are located nearby majority class instances. They learn a mapping from minority instance vectors to "disentangled" representations, making use of mutual information estimators (Belghazi et al, 2018) to push these representations away from the majority class and closer to the minority class.…”

Section: Data Augmentationmentioning

confidence: 99%

A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing

Henning,

Beluch,

Fraser

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Many natural language processing (NLP) tasks are naturally imbalanced, as some target categories occur much more frequently than others in the real world. In such scenarios, current NLP models tend to perform poorly on less frequent classes. Addressing class imbalance in NLP is an active research topic, yet, finding a good approach for a particular task and imbalance scenario is difficult.In this survey, the first overview on class imbalance in deep-learning based NLP, we first discuss various types of controlled and realworld class imbalance. Our survey then covers approaches that have been explicitly proposed for class-imbalanced NLP tasks or, originating in the computer vision community, have been evaluated on them. We organize the methods by whether they are based on sampling, data augmentation, choice of loss function, staged learning, or model design. Finally, we discuss open problems and how to move forward.

show abstract

“…By combining the MI with the k-means algorithm, a clustering-based sampling method, global data distribution weighted synthetic oversampling (GDDSYN) outperforms some other existing methods [24]. Another work on text classification also introduced a successful MI-constrained oversampling mechanism (MISO) that safely and robustly re-embeds challenging samples [135]. The MI-based SMOTE is also widely applied to multiple aspects with various classifiers, including the MI classifier [136], the KNN classifier, and the decision tree classifier [137].…”

Section: Information Theorymentioning

confidence: 99%

Analysis and modification of mutual information-based feature selection methods regarding data imbalance and incompleteness

View full text Add to dashboard Cite

It is a meaningful and unforgettable journey toward my Doctoral degree and a critical time that helps me not only Enhance my research abilities but also gain a fresh perspective on life and rediscover myself."A thousand people, a thousand Hamlets." I believe it works for how we feel about this journey. For me, it is more like a war with myself, in which I would like to overcome resistance to change, be brave, and move on. There was a time when I mistook stability for safety and comfort, but after a while, I realized that stability comes from changes since the essence of life is change. Hence, I sincerely appreciate those who support me and encourage me to be brave and complete this vital journey. Without them, I cannot make any of these advancements.I want to first express my most heartfelt appreciation to Professor Nasser Fard, my advisor, for his unwavering support, guidance, and empathy during my graduate studies. He is always there to offer me support whenever I come up with new research ideas. my research would not have been possible to start, continue, or finish without his support. My foremost gratitude also goes to my committees, Prof. Chun-An Chou, Prof. Wei Xie, and Prof. Keivan Sadeghzadeh. I thank them for their wholehearted endorsement, including helpful suggestions that contributed greatly to this dissertation and valuable insights into my research.I also would like to thank my fellow Ph.D. colleagues in the MIE department for their support and encouragement throughout my graduate studies. I cherish all my memories with them, from the midnight cramming and refreshment breaks to uplifting affirmations and talks.Finally, I would like to thank my family, especially my parents. Their caring nature towards me never ceased despite the distance, and I still remember the warmth I felt every time I received an overseas call from them. They have always been my harbor and have made me brave enough to face all challenges I met in this journey. It is their unconditional backing that has formed the cornerstone of my life.

show abstract

Re-embedding Difficult Samples via Mutual Information Constrained Semantically Oversampling for Imbalanced Text Classification

Cited by 7 publications

References 25 publications

Reducing Sentiment Bias in Pre-trained Sentiment Classification via Adaptive Gumbel Attack

Reducing Sentiment Bias in Pre-trained Sentiment Classification via Adaptive Gumbel Attack

A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing

Analysis and modification of mutual information-based feature selection methods regarding data imbalance and incompleteness

Contact Info

Product

Resources

About