Maximizing Stylistic Control and Semantic Accuracy in NLG: Personality Variation and Discourse Contrast

Abstract

Neural generation methods for task-oriented dialogue typically generate from a meaning representation that is populated using a database of domain information, such as a table of data describing a restaurant. While earlier work focused solely on the semantic fidelity of outputs, recent work has started to explore methods for controlling the style of the generated text while simultaneously achieving semantic accuracy. Here we experiment with two stylistic benchmark tasks, generating language that exhibits variation in personality, and generating discourse contrast. We report a huge performance improvement in both stylistic control and semantic accuracy over the state of the art on both of these benchmarks. We test several different models and show that putting stylistic conditioning in the decoder and eliminating the semantic re-ranker used in earlier models results in more than 15 points higher BLEU for Personality, with a reduction of semantic error to near zero. We also report an improvement from .75 to .81 in controlling contrast and a reduction in semantic error from 16% to 2%.

Returning the N to NLP: Towards Contextually Personalized Classification Models

“…Similar developments are emerging for example in the area of personalized language models (Ji et al, 2019;Wen et al, 2012;Yoon et al, 2017;McMahan et al, 2017), which are largely used e.g. in predictive writing, and in natural language generation (Oraby et al, 2018;Harrison et al, 2019), aiming e.g. at selecting and preserving a consistent personality and style within a discourse.…”

“…a user may prefer a model that correctly interprets her sarcasm even when most annotators typically don't recognize it. We can take inspiration from subjective measures used in evaluating spoken dialogue systems, such as A/B testing (Kohavi et al, 2014), customer satisfaction (Kelly et al, 2009;Kiseleva et al, 2016) or interestingness (Harrison et al, 2019;Oraby et al, 2018).…”

An Evaluation of Disentangled Representation Learning for Texts

“…which is by design constrained to have two separable aspects of variation (meaning and style); this is quite rare in real-world data. Indeed, the best performing style transfer model on this dataset, from Harrison et al (2019), is a heavily supervised one that conditions a seq-2-seq model with annotations for each type of variation in the surface realisations (i.e., the presence of certain tokens).…”

Correcting Texts Generated by Transformers using Discourse Features andWeb Mining

“…Ji et al (2016) suggested a similar approach but predicted discourse relations using RNN. Harrison et al (2019) investigated an approach that allows generating text depending on the need of the "Contrast" relation. One of the main goals was that the model itself should be able to determine which items are suitable for contradistinction and which values are acceptable for them.…”