An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference

Abstract

The prior work on natural language inference (NLI) debiasing mainly targets at one or few known biases while not necessarily making the models more robust. In this paper, we focus on the model-agnostic debiasing strategies and explore how to (or is it possible to) make the NLI models robust to multiple distinct adversarial attacks while keeping or even strengthening the models' generalization power. We firstly benchmark prevailing neural NLI models including pretrained ones on various adversarial datasets. We then try to combat distinct known biases by modifying a mixture of experts (MoE) ensemble method (Clark et al., 2019) and show that it's nontrivial to mitigate multiple NLI biases at the same time, and that model-level ensemble method outperforms MoE ensemble method. We also perform data augmentation including text swap, word substitution and paraphrase and prove its efficiency in combating various (though not all) adversarial attacks at the same time. Finally, we investigate several methods to merge heterogeneous training data (1.35M) and perform model ensembling, which are straightforward but effective to strengthen NLI models.

Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets

“…Biases in NLI Table 5: Results on the NLI adversarial test benchmark (Liu et al, 2020b). We compare with the data augmentation techniques investigated by Liu et al (2020b).…”

“…Biases in NLI Table 5: Results on the NLI adversarial test benchmark (Liu et al, 2020b). We compare with the data augmentation techniques investigated by Liu et al (2020b). * are reported results and underscore indicates statistical significance against the baseline.…”

“…To evaluate our approach, we use the task of Natural Language Inference (NLI), which offers a wide range of datasets (including challenge datasets) for various domains. We generate debiased SNLI (Bowman et al, 2015) and MNLI (Williams et al, 2018) distributions and evaluate the generalisability of models trained on them to out-of-distribution hard evaluation sets (Gururangan et al, 2018;McCoy et al, 2019), and the adversarial attack suite for NLI proposed by Liu et al (2020b). Furthermore, we compare our method to strong debiasing strategies from the literature (Belinkov et al, 2019b;Stacey et al, 2020;Clark et al, 2019;Karimi Mahabadi et al, 2020;Utama et al, 2020;Sanh et al, 2021;Ghaddar et al, 2021).…”

Behind the Scenes: An Exploration of Trigger Biases Problem in Few-Shot Event Classification

“…The trigger bias proposed in our paper belongs to selection bias and model overamplification bias. Bias has also been investigated in natural language inference [1,6,7,13,[21][22][23], question answering [24], ROC story cloze [2,28], lexical inference [17], visual question answering [12], etc. To our best knowledge, we are the first to present the biases in FSEC, i.e., trigger overlapping and trigger separability.…”

Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference

“…Such methods can be roughly categorized into two classes: sentence embedding bottleneck methods which first encode the two sentences as vectors and then feed them into a classifier for classification (Conneau et al, 2017;Nie and Bansal, 2017;Choi et al, 2018;Chen et al, 2017b;Wu et al, 2018), and more general methods which usually involve interactions while encoding the two sentences in the pair (Chen et al, 2017a;Gong et al, 2018;Parikh et al, 2016). Recently, NLI models are shown to be biased towards spurious surface patterns in the human annotated datasets (Poliak et al, 2018;Gururangan et al, 2018;Liu et al, 2020a), which makes them vulnerable to adversarial attacks (Glockner et al, 2018;Minervini and Riedel, 2018;McCoy et al, 2019;Liu et al, 2020b).…”