Stochastic Language Generation in Dialogue using Factored Language Models

Abstract

Most previous work on trainable language generation has focused on two paradigms: (a) using a statistical model to rank a set of pre-generated utterances, or (b) using statistics to determine the generation decisions of an existing generator. Both approaches rely on the existence of a handcrafted generation component, which is likely to limit their scalability to new domains. The first contribution of this article is to present BAGEL, a fully data-driven generation method that treats the language generation task as a search for the most likely sequence of semantic concepts and realization phrases, according to Factored Language Models (FLMs). As domain utterances are not readily available for most natural language generation tasks, a large creative effort is required to produce the data necessary to represent human linguistic variation for nontrivial domains. This article is based on the assumption that learning to produce paraphrases can be facilitated by collecting data from a large sample of untrained annotators using crowdsourcing-rather than a few domain experts-by relying on a coarse meaning representation. A second contribution of this article is to use crowdsourced data to show how dialogue naturalness can be improved by learning to vary the output utterances generated for a given semantic input. Two datadriven methods for generating paraphrases in dialogue are presented: (a) by sampling from the n-best list of realizations produced by BAGEL's FLM reranker; and (b) by learning a structured perceptron predicting whether candidate realizations are valid paraphrases. We train BAGEL on a set of 1,956 utterances produced by 137 annotators, which covers 10 types of dialogue acts and 128 semantic concepts in a tourist information system for Cambridge. An automated evaluation shows that BAGEL outperforms utterance class LM baselines on this domain. A human evaluation of 600 resynthesized dialogue extracts shows that BAGEL's FLM output produces utterances comparable to a handcrafted baseline, whereas the perceptron classifier performs worse. Interestingly, human judges find the system sampling from the n-best list to be more natural than a system always returning the first-best utterance. The judges are also more willing to interact with the n-best system in the future. These results suggest that capturing the large variation found in human language using data-driven methods is beneficial for dialogue interaction.

Sequence-to-Sequence Models for Data-to-Text Natural Language Generation: Word- vs. Character-based Processing and Output Diversity

“…While the first NLG systems relied on handwritten rules or templates that were filled with the input information (Cheyer and Guzzoni, 2006;Mirkovic et al, 2006), the availability of larger datasets has accelerated the progress in statistical methods to train NLG systems from data-text pairs in the last twenty years (Oh and Rudnicky, 2000;Mairesse and Young, 2014). Generating output via language models based on recurrent neural networks (RNNs) conditioned on the input (Sutskever et al, 2011) proved to be an effective method for end-to-end NLG (Wen et al, 2015a(Wen et al, ,b, 2016.…”

The Code2Text Challenge: Text Generation in Source Libraries

“…Recent data-driven approaches in NLG have been successful in modeling end-to-end generation from unaligned input-output, cf. (Angeli et al, 2010;Mairesse and Young, 2014;Dušek and Jurcicek, 2015;Wen et al, 2016). However, these system have been mostly tested on datasets (e.g., in the restaurant domain) that require describing very similar entities, entities that are encoded in MRs that have considerable lexical overlap with the target text output.…”

On the role of linguistic descriptions of data in the building of natural language generation systems

“…Recent approaches cover the use of natural language generation in interactive spoken dialogue systems through models which dynamically adapt to the users' level of expertise [33], a fully data-driven generation method that treats the language generation task as a search for the most likely sequence of semantic concepts and realization phrases [34], and a domain-independent approximation that performs content determination and surface realization in a joint unsupervised fashion through the use of a probabilistic context-free grammar [35].…”