Unsupervised Text Style Transfer with Padded Masked Language Models

Malmi, Eric; Severyn, Aliaksei; Rothe, Sascha

doi:10.18653/v1/2020.emnlp-main.699

Cited by 44 publications

(58 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to comparing end-to-end systems, we also compare LEWIS to concurrent editing and synthesis methods by Malmi et al (2019Malmi et al ( , 2020. Table 2 shows that training the same model (LaserTagger) on our data improves and BLEU by 9.5 (the accuracy difference is not directly comparable since Malmi et al (2020) used a BERT classifier and did not release model output). This suggests that our data synthesis procedure produces higher quality data than Malmi et al (2020).…”

Section: Resultsmentioning

confidence: 99%

LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer

Reid¹,

Zhong²

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Many types of text style transfer can be achieved with only small, precise edits (e.g. sentiment transfer from I had a terrible time... to I had a great time...).We propose a coarse-to-fine editor for style transfer that transforms text using Levenshtein edit operations (e.g. insert, replace, delete). Unlike prior single-span edit methods, our method concurrently edits multiple spans in the source text. To train without parallel style text pairs (e.g. pairs of +/-sentiment statements), we propose an unsupervised data synthesis procedure. We first convert text to style-agnostic templates using style classifier attention (e.g. I had a SLOT time...), then fill in slots in these templates using fine-tuned pretrained language models. Our method outperforms existing generation and editing style transfer methods on sentiment (YELP, AMAZON) and politeness (POLITE) transfer. In particular, multi-span editing achieves higher performance and more diverse output than single-span editing. Moreover, compared to previous methods on unsupervised data synthesis, our method results in higher quality parallel style pairs and improves model performance. 1

show abstract

Section: Resultsmentioning

confidence: 99%

LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer

Reid¹,

Zhong²

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

show abstract

“…Finally, by allowing control through control codes and [BLANK]s, Polyjuice supports humangenerator collaboration, where a person specifies desired changes (e.g., perturb the sentence subject). Such collaboration is hard to imagine using automatic generators with no control, or with coarser control through predefined style attributes or labels (Madaan et al, 2020;Malmi et al, 2020). To our knowledge, prior work on controlled generation (Keskar et al, 2019;Dathathri et al, 2020) does not address counterfactual generation.…”

Section: Related Workmentioning

confidence: 99%

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models

Wu¹,

Ribeiro²,

Heer³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts.

show abstract

“…However, they are recently criticized for resulting in poor content preservation (Xu et al, 2018;Jin et al, 2019;Subramanian et al, 2018) and alternatively, translation-based models are proposed that use reconstruction and back-translation losses (e.g., Logeswaran et al (2018); Prabhumoye et al (2018)). Another line of work, focuses on manipulation methods that remove the style-specific attribute of text (e.g., ; Xu et al (2018)), while recent approaches use reinforcement learning (e.g., ; Gong et al (2019), probabilistic formulations (He et al, 2020), and masked language models (Malmi et al, 2020).…”

Section: Related Workmentioning

confidence: 99%