Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators

Abstract

Encoder-decoder based neural architectures serve as the basis of state-of-the-art approaches in end-to-end open domain dialog systems. Since most of such systems are trained with a maximum likelihood (MLE) objective they suffer from issues such as lack of generalizability and the generic response problem, i.e., a system response that can be an answer to a large number of user utterances, e.g., "Maybe, I don't know." Having explicit feedback on the relevance and interestingness of a system response at each turn can be a useful signal for mitigating such issues and improving system quality by selecting responses from different approaches. Towards this goal, we present a system that evaluates chatbot responses at each dialog turn for coherence and engagement. Our system provides explicit turn-level dialog quality feedback, which we show to be highly correlated with human evaluation. To show that incorporating this feedback in the neural response generation models improves dialog quality, we present two different and complementary mechanisms to incorporate explicit feedback into a neural response generation model: reranking and direct modification of the loss function during training. Our studies show that a response generation model that incorporates these combined feedback mechanisms produce more engaging and coherent responses in an open-domain spoken dialog setting, significantly improving the response quality using both automatic and human evaluation.

Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems

“…Eventually, as Table 2 demonstrates, the mean κ agreement and mean Pearson correlation between evaluators participating in our experiments were 0.52 and 0.93. In the context of dialogue system evaluation where agreement is usually quite low (Venkatesh et al 2018;Ghandeharioun et al 2019;Yi et al 2019), these numbers show relatively high agreement between annotators. This provides evidence that engagement can be measured not only at the conversation level but also at the utterance level.…”

Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems

“…Likability quantifies how much a set of one or more qualities makes a response more likable for a particular task. These qualities can be diversity (Li et al, 2016), sentiment (Rashkin et al, 2019), specificity (Ke et al, 2018), engagement (Yi et al, 2019), fluency (Kann et al, 2018) and more. A likable response may or may not be sensible to the context.…”

HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

“…In this way, dialog systems could detect and react to user's disengagement in both open-domain dialogs (Yu et al, 2016) and taskoriented dialogs (Yu et al, 2017). During training, our model could also be used as real-time feedback to benefit dialog policy learning (Yi et al, 2019). Second, HERALD could quantify user engagement and be used as an automatic dialog evaluation metric.…”