Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification

Pruksachatkun, Yada; Krishna, Satyapriya; Dhamala, Jwala; Gupta, Rahul; Chang, Kai-Wei

doi:10.18653/v1/2021.findings-acl.294

Cited by 13 publications

(16 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where ∆ TPR (∆ FPR ) is the true positive rate (false negative rate) for sensitive attribute a and TPR overall (FPR overall ) is the overall true positive rate (false negative rate). Following (Pruksachatkun et al, 2021), we define equalized odds gap ∆ EO = ∆ TPR + ∆ FPR , since equalized odds aligns with ∆ TPR + ∆ FPR , and when it is satisfied, ∆ TPR = ∆ FPR = 0 (Borkan et al, 2019). Note that when |Y| > 2, ∆ EO will be summed over each value in Y since TPR and FPR are defined over each class (i.e., ∆ EO = y ∆ y EO ).…”

Section: Methodsmentioning

confidence: 99%

Conditional Supervised Contrastive Learning for Fair Text Classification

Chi¹,

Shand²,

Yu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data. However, the learned representations could potentially lead to performance disparities in downstream tasks, such as increased silencing of underrepresented groups in toxicity comment classification. In light of this challenge, in this work, we study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with fairness constraint and conditional supervised contrastive objectives, and then propose to use conditional supervised contrastive objectives to learn fair representations for text classification. We conduct experiments on two text datasets to demonstrate the effectiveness of our approaches in balancing the trade-offs between task performance and bias mitigation among existing baselines for text classification. Furthermore, we also show that the proposed methods are stable in different hyperparameter settings.

show abstract

Section: Methodsmentioning

confidence: 99%

Conditional Supervised Contrastive Learning for Fair Text Classification

Chi¹,

Shand²,

Yu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This task is more difficult within the NLP domain due to the discrete nature of text, but several works (Alzantot et al, 2018;Zhang et al, 2020) have proven successful at inducing model errors. Real-world use of NLP requires resilience to such attacks and our work complements robust training (Parvez et al, 2018) and robust certification (Ye et al, 2020;Pruksachatkun et al, 2021) to produce more reliable models.…”

Section: Related Workmentioning

confidence: 99%

Sibylvariant Transformations for Robust Text Classification

Harel-Canada¹,

Gulzar²,

Peng³

et al. 2022

Preprint

View full text Add to dashboard Cite

The vast majority of text transformation techniques in NLP are inherently limited in their ability to expand input space coverage due to an implicit constraint to preserve the original class label. In this work, we propose the notion of sibylvariance (SIB) to describe the broader set of transforms that relax the labelpreserving constraint, knowably vary the expected class, and lead to significantly more diverse input distributions. We offer a unified framework to organize all data transformations, including two types of SIB: (1) Transmutations convert one discrete kind into another, (2) Mixture Mutations blend two or more classes together. To explore the role of sibylvariance within NLP, we implemented 41 text transformations, including several novel techniques like Concept2Sentence and SentMix.Sibylvariance also enables a unique form of adaptive training that generates new input mixtures for the most confused class pairs, challenging the learner to differentiate with greater nuance. Our experiments on six benchmark datasets strongly support the efficacy of sibylvariance for generalization performance, defect detection, and adversarial robustness.

show abstract

“…And the relevant keywords which categorise the transformation. t/evaluation data splits allows for testing the robustness of models and for identifying possible biases; on the other hand, applying transformations and filters to training data (data augmentation) allows for possibly mitigating the detected robustness and bias issues (Wang et al, 2021b;Pruksachatkun et al, 2021;Si et al, 2021).…”

Section: Format Of a Transformationmentioning

confidence: 99%

“…The resultant tokens are also assigned new tags. Exploiting this transformation has shown to empirically benefit named entity tagging (Yaseen and Langer, 2021) and hence could arguably benefit other lowresource tagging tasks (Bhatt and Dhole, 2020;Khachatrian et al, 2019;Gupta et al, 2021).…”

Section: B9 Backtranslation For Named Entity Recognitionmentioning

confidence: 99%

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Dhole¹,

Gangal²,

Gehrmann³

et al. 2021

Preprint

View full text Add to dashboard Cite

Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained on. In this paper, we present NL-Augmenter, a new participatory Pythonbased natural language augmentation framework which supports the creation of both transformations (modifications to the data) and filters (data splits according to specific features). We describe the framework and an initial set of 117 transformations and 23 filters for a variety of natural language tasks. We demonstrate the efficacy of NL-Augmenter by using several of its tranformations to analyze the robustness of popular natural language models. The infrastructure, datacards and robutstness analysis results are available publicly on the NL-Augmenter repository (https://github. com/GEM-benchmark/NL-Augmenter).

show abstract

Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification

Cited by 13 publications

References 38 publications

Conditional Supervised Contrastive Learning for Fair Text Classification

Conditional Supervised Contrastive Learning for Fair Text Classification

Sibylvariant Transformations for Robust Text Classification

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Contact Info

Product

Resources

About