What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation

Abstract

Accurate automatic evaluation metrics for open-domain dialogs are in high demand. Existing model-based metrics for system response evaluation are trained on human annotated data, which is cumbersome to collect. In this work, we propose to use information that can be automatically extracted from the next user utterance, such as its sentiment or whether the user explicitly ends the conversation, as a proxy to measure the quality of the previous system response. This allows us to train on a massive set of dialogs with weak supervision, without requiring manual system turn quality annotations. Experiments show that our model is comparable to models trained on human annotated data. Furthermore, our model generalizes across both spoken and written opendomain dialog corpora collected from real and paid users.

Improving Open-Domain Dialogue Evaluation with a Causal Inference Model

“…Another idea is to score a proposed system utterance by the probability that it elicits a particular user response type, such as disinterest or criticism [46,31,49,12].…”

“…For example, the FED framework scores proposed utterances by utilizing the Di-aloGPT LM probabilities for the subsequent user utterances such as "That is interesting" [56,31]. Predictive models can also be generated from large dialogue corpora by using the following user utterance as weak supervision [12,46].…”

“…Many recent approaches work with embedding features, such as sentiment scores and utterance classification, before predicting the user rating [53,14]. Sentiment analysis [21,12] and dialogue features analysis [30,42,49] methods, for instance, show solid results for predicting user-assigned ratings.…”

“…Next, the ODES categories assign the treatment T for the CF-LSTM. We also extract 3 different types of utterance sentiment features: sentiment valence, satisfaction, and activation, which are highly correlated with user ratings [21,12].…”

“…For task-oriented dialogues, frameworks such as Paradise [51] model the relationship between user satisfaction, task completion, and cost factors such as dialogue length, word error rate, and dialogue behaviors. However, open-domain dialogue systems such as those built for the Alexa Prize SocialBot Grand Challenge [36,9], where there is no clearly defined task, require new metrics and methods for evaluation that better reflect their affordances [49,21,13,12,16,47].…”