Tailor: Generating and Perturbing Text with Semantic Controls

Abstract

Making controlled perturbations is essential for various tasks (e.g., data augmentation), but building task-specific generators can be expensive. We introduce Tailor, a taskagnostic generation system that perturbs text in a semantically-controlled way. With unlikelihood training, we design Tailor's generator to follow a series of control codes derived from semantic roles. Through modifications of these control codes, Tailor can produce fine-grained perturbations. We implement a set of operations on control codes that can be composed into complex perturbation strategies, and demonstrate their effectiveness in three distinct applications: First, Tailor facilitates the construction of high-quality contrast sets that are lexically diverse, and less biased than original task test data. Second, paired with automated labeling heuristics, Tailor helps improve model generalization through data augmentation: We obtain an average gain of 1.73 on an NLI challenge set by perturbing just ≈5% of training data. Third, without any finetuning overhead, Tailor's perturbations effectively improve compositionality in fine-grained style transfer, outperforming fine-tuned baselines on 6 transfers.

Retrieval-guided Counterfactual Generation for QA

“…we show examples of semantic changes that occur in our data, which also occur in prior work (Gardner et al, 2020;Ross et al, 2021b;, including reference changes, predicate changes, negations, syntactic changes, question expansions, disambiguations, and question contractions. Note that we do not use any specialized semantic framework or structured representations to achieve the same type of transformation.…”

“…Our method generates highly diverse counterfactuals covering a range of semantic phenomena ( §4), including many of those found in existing work that relies on meaning representation pivots (Ross et al, 2021b;Geva et al, 2021) or human generation (Bartolo et al, 2020;Gardner et al, 2020). Compared to alternative sources of synthetic data ( §5.1), training augmented with RGF data leads to increased performance on a variety of settings ( §5.2, §5.3), including out-of-domain (Fisch et al, 2019) and contrast evaluation sets (Bartolo et al, 2020;Gardner et al, 2020), while maintaining indomain performance.…”

“…Webster et al (2020) use templates to create large-scale counterfactual data for pre-training to reduce gender bias. More recent work has specifically focused on using meaning representation frameworks and structured inputs to automatically perturb inputs -grammar formalisms , semantic role labeling (Ross et al, 2021b), structured image representations like scene graphs (Bitton et al, 2021) and query decompositions in multi-hop reasoning datasets (Geva et al, 2021). (Ye et al, 2021) and (Longpre et al, 2021) per-turb contexts instead of questions by swapping out all mentions of a named entity.…”

“…Following desiderata from and Ross et al (2021b), we evaluate our RGF data along four qualitative evaluation measures: fluency, correctness, minimality, and directionality.…”

“…Unlike many counterfactual generation methods, RGF natively creates fully-labeled (q , c , a ) exam- (Ross et al, 2021b) C: When did lebron_james come into the league ?…”