Retrieval-guided Counterfactual Generation for QA

Paranjape, Bhargavi; Lamm, Matthew; Tenney, Ian

doi:10.18653/v1/2022.acl-long.117

Cited by 8 publications

(5 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Counterfactual Inference. Our work is based on counterfactual inference (Pearl et al, 2000), which has shown promising results in various NLP tasks, including question answering (Paranjape et al, 2022;, machine translation and story generation (Qin et al, 2019;. In particular, uses counterfactual inference for response generation, which explores potential responses via counterfactual off-policy training.…”

Section: Further Discussionmentioning

confidence: 99%

Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues

Ou¹,

Zhang²,

Feng³

et al. 2022

Preprint

View full text Add to dashboard Cite

The construction of open-domain dialogue systems requires high-quality dialogue datasets. The dialogue data admits a wide variety of responses for a given dialogue history, especially responses with different semantics. However, collecting high-quality such a dataset in most scenarios is labor-intensive and timeconsuming. In this paper, we propose a data augmentation method to automatically augment high-quality responses with different semantics by counterfactual inference. Specifically, given an observed dialogue, our counterfactual generation model first infers semantically different responses by replacing the observed reply perspective with substituted ones. Furthermore, our data selection method filters out detrimental augmented responses. Experimental results show that our data augmentation method can augment high-quality responses with different semantics for a given dialogue history, and can outperform competitive baselines on multiple downstream tasks.

show abstract

Section: Further Discussionmentioning

confidence: 99%

Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues

Ou¹,

Zhang²,

Feng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In NLP, early approaches (Kaushik et al, 2020) pair each instance with its label-flipped augmentation, obtained from perturbing labels by human efforts with ϵ lexical changes. Later approaches propose automatic syntheses to alleviate human efforts (Han et al, 2021;Paranjape et al, 2022;Calderon et al, 2022).…”

Section: A4 Counterfactual Text Augmentationmentioning

confidence: 99%

Intervention-Based Alignment of Code Search with Execution Feedback

Han,

Kim,

Hwang

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

One of the fundamental goals in code search is to retrieve a functionally correct code for a given natural language query. As annotating for correctness requires executing test cases (i.e. obtaining execution feedback), existing code search training datasets approximate text-code co-occurrences as positive execution feedback. However, this approximation may misalign models' retrieval decisions from ground-truth correctness. To address such limitation, we propose Code Intervention-based Reinforcement Learning (CIRL) that perturbs training code to result in misalignment (i.e. code intervention), then tests models' decisions and corrects them with the execution feedback by reinforcement learning. The first technical contribution of CIRL is to induce the execution feedback from perturbation, without actual execution. Secondly, CIRL introduces structural perturbations using abstract syntax trees, going beyond simple lexical changes. Experimental results on various datasets demonstrate the effectiveness of CIRL compared to conventional approaches.

show abstract

“…However, Joshi and He (2022) find that a limited set of perturbation types further exacerbates biases, resulting in poor generalization to unseen perturbation types. Generally, creating an assorted set of instance-specific perturbations is challenging, often requiring external knowledge (Paranjape et al, 2022). Retrieval Augmented Generation Retrieving task-relevant knowledge from a large corpus of unstructured and unlabeled text has proven to be very effective for knowledge-intensive language generation tasks like question answering , machine translation (Gu et al, 2018) and dialogue generation (Weston et al, 2018).…”

Section: Related Workmentioning

confidence: 99%

“…In a similar vein, CORE uses learned retrieval for counterfactual generation. While Paranjape et al (2022) use off-the-shelf retrieval models to generate counterfactuals for QA, learning to retrieve counterfactuals is non-trivial for problems other than QA. CORE provides a recipe to train retrieval for general tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Contrast sets (Gardner et al, 2020) and counterfactual data (Kaushik et al, 2020) provide minimal input perturbations that change model predictions, and serve as an effective means to evaluate brittleness to out-of-distribution data (Wang et al, 2021). Counterfactual data augmentation (CDA) has shown to improve model robustness to OOD data and input perturbations (Geva et al, 2021;Wu et al, 2021;Paranjape et al, 2022;. Alternate methods like debiasing data (Wu 1 Code at https://github.com/tanay2001/CORE Control activities happen everywhere in the entity.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation

Tanay¹,

Paranjape²,

Hajishirzi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Counterfactual data augmentation (CDA)i.e., adding minimally perturbed inputs during training -helps reduce model reliance on spurious correlations and improves generalization to out-of-distribution (OOD) data. Prior work on generating counterfactuals only considered restricted classes of perturbations, limiting their effectiveness.We present COunterfactual Generation via Retrieval and Editing (CORE), a retrieval-augmented generation framework for creating diverse counterfactual perturbations for CDA. For each training example, CORE first performs a dense retrieval over a task-related unlabeled text corpus using a learned bi-encoder and extracts relevant counterfactual excerpts. CORE then incorporates these into prompts to a large language model with few-shot learning capabilities, for counterfactual editing. Conditioning language model edits on naturally occurring data results in diverse perturbations. Experiments on natural language inference and sentiment analysis benchmarks show that CORE counterfactuals are more effective at improving generalization to OOD data compared to other DA approaches. We also show that the CORE retrieval framework can be used to encourage diversity in manually authored perturbations 1 .

show abstract

Retrieval-guided Counterfactual Generation for QA

Cited by 8 publications

References 28 publications

Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues

Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues

Intervention-Based Alignment of Code Search with Execution Feedback

CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation

Contact Info

Product

Resources

About