Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer

Narasimhan, Sharan; Dey, Suvodip; Desarkar, Maunendra Sankar

doi:10.18653/v1/2022.naacl-main.34

Cited by 3 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work focuses on various common paradigms such as disentanglement (Hu et al, 2017;Shen et al, 2017), cycle-consistency losses (Yi et al, 2020;Luo et al, 2019;Dai et al, 2019;Liu et al, 2021), induction (Narasimhan et al, 2022;Shen et al, 2020). Jin et al (2021) and Hu et al (2020) provide surveys detailing the current state of style transfer and lay down useful taxonomies to structure the field.…”

Section: Related Workmentioning

confidence: 99%

On Text Style Transfer via Style-Aware Masked Language Models

Narasimhan,

Dey

et al. 2023

Proceedings of the 16th International Natural Language Generation Conference

View full text Add to dashboard Cite

Text Style Transfer (TST) involves transforming a source sentence with a given style label to an output with another target style meanwhile preserving content and fluency. We look at a fill-in-the-blanks approach (also referred to as prototype editing), where the source sentence is stripped off all style-containing words and filled in with suitable words. This closely resembles a Masked Language Model (MLM) objective, with the added initial step of masking only relevant style words rather than BERT's random masking. We show this simple MLM, trained to reconstruct style-masked sentences back into their original style, can even transfer style by making this MLM "Style-Aware". This simply involves appending the source sentence with a target style special token. The Style-Aware MLM (SA-MLM) now also accounts for the direction of style transfer and enables style transfer by simply manipulating these special tokens. To learn this n-word to n-word style reconstruction task, we use a single transformer encoder block with 8 heads, 2 layers and no auto-regressive decoder, making it non-generational. We empirically show that this lightweight encoder trained on a simple reconstruction task compares with elaborately engineered state-of-the-art TST models for even complex styles like Discourse or flow of logic, i.e. Contradiction to Entailment and vice-versa. Additionally, we introduce a more accurate attention-based style-masking step and a novel "attention-surplus" method to determine the position of masks from any arbitrary attribution model in O(1) time. Finally, we show that the SA-MLM arises naturally by considering a probabilistic framework for style transfer.

show abstract

Section: Related Workmentioning

confidence: 99%

On Text Style Transfer via Style-Aware Masked Language Models

Narasimhan,

Dey

et al. 2023

Proceedings of the 16th International Natural Language Generation Conference

View full text Add to dashboard Cite

show abstract

“…Recent work focuses on various common paradigms such as disentanglement , cycle-consistency losses Luo et al, 2019;, induction (Narasimhan et al, 2022;. Jin et al (2021) and provide surveys detailing the current state of style transfer and lay down useful taxonomies to structure the field.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 16th International Natural Language Generation Conference

2023

View full text Add to dashboard Cite

In this paper, we introduce a new beam search algorithm that improves the generalization of neural generators to unseen examples, especially in low-resource data-to-text settings. Our algorithm aims to reduce the number of omissions and hallucinations during the decoding process. For this purpose, it relies on two regression models to explicitly characterize factual errors. We explain how to create a new dataset to train these models given an original training set of less than a thousand data points. We apply our approach in the low-resource, legal setting using the French Plum2Text dataset, as well as in English using WebNLG. We observe in our experiment that this combination improves the faithfulness of pre-trained neural text generators using both human and automatic evaluation. Moreover, our approach offers a level of interpretability by predicting the number of omissions and hallucinations present in a given generation with respect to the input data. Finally, we visualize our algorithm's exploration of the hypothesis space at different steps during the decoding process.

show abstract

“…the style-transferred output is not available. Past work focuses on various common paradigms such as disentanglement (Hu et al, 2017;Shen et al, 2017), cycle-consistency losses (Yi et al, 2020;Luo et al, 2019;Dai et al, 2019;Liu et al, 2021), induction (Narasimhan et al, 2022;Shen et al, 2020) etc. We focus on a sentence to sentence "transduction" (or prototype editing) method, a solution which naturally emerges when following a probabilistic formulation consisting of a single transduction model with a latent prior consisting of a style-absent corpus.…”

Section: Introductionmentioning

confidence: 99%

On Text Style Transfer via Style Masked Language Models

Narasimhan¹,

Shekar²,

Dey³

et al. 2022

Preprint

View full text Add to dashboard Cite

Text Style Transfer (TST) is performable through approaches such as latent space disentanglement, cycle-consistency losses, prototype editing etc. The prototype editing approach, which is known to be quite successful in TST, involves two key phases a) Masking of source style-associated tokens and b) Reconstruction of this source-style masked sentence conditioned with the target style. We follow a similar transduction method, in which we transpose the more difficult direct source to target TST task to a simpler Style-Masked Language Model (SMLM) Task, wherein, similar to BERT (Devlin et al., 2019a), the goal of our model is now to reconstruct the source sentence from its style-masked version. We arrive at the SMLM mechanism naturally by formulating prototype editing/ transduction methods in a probabilistic framework, where TST resolves into estimating a hypothetical parallel dataset from a partially observed parallel dataset, wherein each domain is assumed to have a common latent style-masked prior. To generate this style-masked prior, we use "Explainable Attention" as our choice of attribution for a more precise style-masking step and also introduce a cost-effective and accurate "Attribution-Surplus" method of determining the position of masks from any arbitrary attribution model in O(1) time. We empirically show that this non-generational approach well suites the "content preserving" criteria for a task like TST, even for a complex style like Discourse Manipulation. Our model, the Style MLM, outperforms strong TST baselines and is on par with state-of-the-art TST models, which use complex architectures and orders of more parameters.* A key difference to BERT, ours is not using a selfsupervised approach of pre-training over large corpora and fine-tuning for a downstream task.

show abstract

Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer

Cited by 3 publications

References 17 publications

On Text Style Transfer via Style-Aware Masked Language Models

On Text Style Transfer via Style-Aware Masked Language Models

Proceedings of the 16th International Natural Language Generation Conference

On Text Style Transfer via Style Masked Language Models

Contact Info

Product

Resources

About