Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study

Abstract

Neural generative models have been become increasingly popular when building conversational agents. They offer flexibility, can be easily adapted to new domains, and require minimal domain engineering. A common criticism of these systems is that they seldom understand or use the available dialog history effectively. In this paper, we take an empirical approach to understanding how these models use the available dialog history by studying the sensitivity of the models to artificially introduced unnatural changes or perturbations to their context at test time. We experiment with 10 different types of perturbations on 4 multi-turn dialog datasets and find that commonly used neural dialog architectures like recurrent and transformer-based seq2seq models are rarely sensitive to most perturbations such as missing or reordering utterances, shuffling words, etc. Also, by open-sourcing our code, we believe that it will serve as a useful diagnostic tool for evaluating dialog systems in the future 1 .

Ranking Enhanced Dialogue Generation

“…Results show that all of the REDfull models get larger PPL increases in most kinds of perturbations than origin models and thus more sensitive to history utterance perturbations. It proves that the dynamic (order) information is more e ectively used by RED models according to the premise in [21] that the more sensitive the model to perturbations, the stronger ability for it of modeling dynamics.…”

“…We conduct experiments on three multi-turn dialogue datasets with di erent styles, they are the bAbI dialog [4], the PersonaChat [35] and the Chinese customer service dataset (JDC) [34] respectively. Each dataset is split into train/valid/test sets according to the previous works [21,34]. Note that each multi-turn dialogue in the three datasets is processed to many history-response pairs with di erent history lengths.…”

“…These model structures proposed for the sequential data assume to capture the dynamics implicitly. Nevertheless, Sankar et al [21] point out that neither the recurrent neural network nor the Transformer neural network in the Seq2Seq framework fully captures the dynamic ow in the dialogue history. In their study, authors make the perturbations to the dialogue history, but surprisingly nd a marginal hurt to the generation performance.…”

“…The above three models are provided by the ParlAI [18] framework which is identical to the experiments in Sankar's empirical study [21]. The hyperparameters are exactly the same as which provided by their work for fair comparison on the bAbI and Per-sonaChat.…”

“…What's more, we also do utterance perturbation experiments on the PersonaChat test set as implemented in Sankar's empirical study [21]. We rst get the PPL for di erent models on the test set.…”