Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management

Abstract

Task-oriented dialogue systems typically rely on large amounts of high-quality training data or require complex handcrafted rules. However, existing datasets are often limited in size con- sidering the complexity of the dialogues. Additionally, conventional training signal in- ference is not suitable for non-deterministic agent behavior, namely, considering multiple actions as valid in identical dialogue states. We propose the Conversation Graph (ConvGraph), a graph-based representation of dialogues that can be exploited for data augmentation, multi- reference training and evaluation of non- deterministic agents. ConvGraph generates novel dialogue paths to augment data volume and diversity. Intrinsic and extrinsic evaluation across three datasets shows that data augmentation and/or multi-reference training with ConvGraph can improve dialogue success rates by up to 6.4%.

Structure Extraction in Task-Oriented Dialogues with Slot Clustering

“…The extra training data is coherent logically but creates more variety in surface formats, thus provides a significant performance boost for end-to-end response generation. The proposed Multi-Response Data Augmentation (MRDA) beats recent work (Gritta et al, 2021) using Most Frequent Sampling in a single-turn setting without annotated states.…”

“…Then we enable this dictionary to create additional data during training, which allows a language model to learn a balanced distribution. In the following sections, we will briefly introduce the task of single-turn dialogue response generation, the baseline augmentation approach called Most Frequent Sampling (Gritta et al, 2021), and the proposed Multi-Response Data Augmentation.…”

“…Shi et al (2019) and use the learned structures for dialogue policy learning. Gritta et al (2021) augment training data with the proposed Most Frequent Sampling (MFS) to improve the success rate of task-oriented dialog systems. Zhang et al (2020a) propose a Multi-Action Data Augmentation (MADA) framework guiding the dialog policy to learn a balanced action distribution.…”

“…Gritta et al (2021) proposed Most Frequent Sampling (MFS) as a data augmentation strategy based on the annotated conversational graph. MFS generates novel training instances so that the most frequent agent actions are preceded by new histories, which is one or more original paths leading to common actions.…”

N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking

“…CoCo (Li et al, 2021) trains a conditional user-utterance generation model, then generates synthetic turns by modifying belief states using a rule-based system and conditioning the model on the modified belief state. Gritta et al (2021) create a working graph of TOD datasets where each edge is a dialogue act and create synthetic dialogues by traversing alternative paths; however, their framework requires user acts to work with. Critically, none of the above techniques exploit the belief state annotations of TODs within an n-shot scenario.…”