StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer

Abstract

Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, STYLEPTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on STYLEPTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, STYLEPTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation.

Tailor: Generating and Perturbing Text with Semantic Controls

“…Here, we show how Tailor can be applied to style transfer. We evaluate Tailor without any finetuning 13 on the StylePTB benchmark (Lyu et al, 2021), which builds on the Penn Treebank and assesses fine-grained stylistic changes (lexical, syntactic, semantic, and thematic), as well as compositions of multiple transfers editing an input sentence along one fine-grained stylistic dimension (e.g., To Future Tense). Compositional transfers require editing along multiple stylistic dimensions at the same time (e.g., To Future Tense+ Active To Passive).…”

“…We compare Tailor with multiple baselines reported by Lyu et al (2021): GPT-2 and Re-trieveEdit are the best-performing single-transfer models evaluated but require separate models to be trained for each individual transfer. CS-GPT-* are models trained on compositional subsets of data (e.g.,, Tense+Voice, detailed in Table 7 caption).…”

“…14 Table 7 shows mean BLEU scores of generations for single and compositional transfers. 15 We evaluate on transfers for which Lyu et al (2021) show model results in the paper, excluding some for which our semantic-role-derived inputs are not well-suited (see §E). When perturbations using Tailor result in unsuccessful transfers, either due to a failure of perturbation strategy (e.g., no verbs are found by our SRL predictor) or due to a degenerate output (see §9), we treat them as having a BLEU score of 0.0; we also show results on a subset of the StylePTB test set that filters out these bad generations (shown in Table 7 as Filtered Test).…”

“…Transfers Evaluated. We evaluate on the transfers in StylePTB for which Lyu et al (2021) report results, as their baselines require training separate models for each transfer. Within this subset of transfers, we exclude PP Back to Front and Passive to Active from evaluation, as they contain < 5 test inputs.…”

“…2) We show that augmenting training data with just a small ratio of Tailor perturbations (≈5%) improves the robustness of Natural Language Inference (NLI) models to inference heuristics, increasing performance on the HANS evaluation set by an average of 1.73 points (McCoy et al, 2019). 3) Without any finetuning, Tailor achieves impressive performance on fine-grained and compositional style transfers ( §7) in the StylePTB benchmark (Lyu et al, 2021), even outperforming models trained on the dataset on 6 transfers.…”