Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning

Abstract

Typical methods for unsupervised text style transfer often rely on two key ingredients: 1) seeking the explicit disentanglement of the content and the attributes, and 2) troublesome adversarial learning. In this paper, we show that neither of these components is indispensable. We propose a new framework that utilizes the gradients to revise the sentence in a continuous space during inference to achieve text style transfer. Our method consists of three key components: a variational auto-encoder (VAE), some attribute predictors (one for each attribute), and a content predictor. The VAE and the two types of predictors enable us to perform gradient-based optimization in the continuous space, which is mapped from sentences in a discrete space, to find the representation of a target sentence with the desired attributes and preserved content. Moreover, the proposed method naturally has the ability to simultaneously manipulate multiple fine-grained attributes, such as sentence length and the presence of specific words, when performing text style transfer tasks. Compared with previous adversarial learning based methods, the proposed method is more interpretable, controllable and easier to train. Extensive experimental studies on three popular text style transfer tasks show that the proposed method significantly outperforms five state-of-the-art methods.

Gradient-guided Unsupervised Text Style Transfer via Contrastive Learning

“…(3) Decode the altered latent representation and get the transferred target with the auto-encoder. The most universal line to acquire the gradient is to train a classifier which outputs the probability of each class directly (Nguyen et al 2017;Wang, Hua, and Wan 2019;Liu et al 2020). The classifier is trained by minimizing:…”

“…We conduct comprehensive comparison with previous stateof-the-art models, including CrossAlign (Shen et al 2017), StyleEmb (Fu et al 2018), MultiDec (Fu et al 2018), Rule-Base (Li et al 2018), DelRetrGen (Li et al 2018), Con-tiSpace (Liu et al 2020) and GBT (Wang, Hua, and Wan 2019). We consider three variants of our model:…”

“…Several efforts have been devoted on the gradient-guided optimization based models, e.g., (Wang, Hua, and Wan 2019;Liu et al 2020), which are usually consist of two subcomponents: (1) Auto-Encoder, which learns the mapping between a discrete sentence space and a continuous latent space; (2) Style classifier, which predicts the style type of a decoded sentence based on its latent representation. The representation of the original sentence is edited iteratively to the target one during inference, along with the direction of the gradient obtained from the style classifier.…”

“…Content invariance is of crucial importance to evaluate the success of a text style transfer model, however, nearly none of existing works takes account of an explicit constraint to ensure the content invariance before and after conversion, which may result in a great discrepancy of the content between the original sentence and the target one. Later, there are various attempts on content consistency, for example, Liu et al (2020) propose a content predictor to tackle such problem by predicting the word features (i.e., Bag-of-Words) of the generated sentence during inference. Nevertheless, experiments illustrate that such method result in trivial improvement.…”

Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization

“…Instead of disentangling content and style, other papers focus on revising an entangled representation of an input. A few previous studies utilize a pre-trained classifier and edit entangled latent variable until it contains target style using the gradientbased optimization (Wang et al, 2019;Liu et al, 2020). He et al (2020) view each domain of data as a partially observable variable, and transfer sentence using amortized variational inference.…”