Adapting Language Models for Non-Parallel Author-Stylized Rewriting

Abstract

Given the recent progress in language modeling using Transformer-based neural models and an active interest in generating stylized text, we present an approach to leverage the generalization capabilities of a language model to rewrite an input text in a target author's style. Our proposed approach adapts a pre-trained language model to generate author-stylized text by fine-tuning on the author-specific corpus using a denoising autoencoder (DAE) loss in a cascaded encoder-decoder framework. Optimizing over DAE loss allows our model to learn the nuances of an author's style without relying on parallel data, which has been a severe limitation of the previous related works in this space. To evaluate the efficacy of our approach, we propose a linguistically-motivated framework to quantify stylistic alignment of the generated text to the target author at lexical, syntactic and surface levels. The evaluation framework is both interpretable as it leads to several insights about the model, and self-contained as it does not rely on external classifiers, e.g. sentiment or formality classifiers. Qualitative and quantitative assessment indicates that the proposed approach rewrites the input text with better alignment to the target style while preserving the original content better than state-of-the-art baselines.

Incorporating Stylistic Lexical Preferences in Generative Language Models

“…We believe that our work can also lead to rewriting of the input content tailored to certain characteristics, if we can design additional rewards to retain content. We have not performed a complete human evaluation due to the high-level of required exper-tise among the annotators for this task, as pointed by Syed et al (2020). Designing the feedback mechanism for such a human evaluation is nontrivial and has been left as a part of the future work along with designing rewarding schemes to capture other author-specific characteristics (e.g., syntactic choices, discourse structure).…”

“…An author's choices of words in these categories define their lexical style. For example, Rudyard Kipling, known for classics of children's literature, had a higher tendency to use more concrete words (like, gongs, rockets, torch) unlike Abraham Lincoln, who being a political writer, used more abstract words (like freedom, patriotism) (Verma and Srinivasan, 2019;Syed et al, 2020). Since an author's style is an amalgam of preferences along these dimensions, our goal is to ensure simultaneous alignment to these multi-dimensional lexical preferences of an author.…”

“…With recent advances in unconstrained language generation (Radford et al, a,b;Brown et al, 2020), an emerging direction is to adapt such pre-trained language models to follow certain stylistic constraints (Wang et al, 2019;Syed et al, 2020). These approaches rely on the inherent properties of the training corpus to tailor generation to target characteristics; for example, implicitly learning author-stylized text generation by training on author-specific corpus (Syed et al, 2020) and learning to generate formal text (Wang et al, 2019). However, it is desirable to have explicit control over certain stylistic aspects in such generation, for e.g., emulating lexical choices of an author in a generation, capturing syntactic constructs, inducing sentential preferences (active vs. passive) in generation.…”

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts

“…(3) approaches with multiple style-specific decoders (Syed et al, 2019;Chen et al, 2019). We highlight several applications including personabased dialogue generation (Li et al, 2016) and creative writing Tikhonov and Yamshchikov, 2018;Vechtomova et al, 2018).…”

Substance over Style: Document-Level Targeted Content Transfer

“…Prior work on controlled generation guides the output of a model using attribute classifiers (Dathathri et al, 2020) or control codes (Keskar et al, 2019), but we find that these models do not perform well on our transfer task ( §4.1.2). In contrast, models built for the transfer task are generally trained at the sentence level (Hu et al, 2017b,a;Li et al, 2018;Rao and Tetreault, 2018;Syed et al, 2019).…”