Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

Abstract

Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly referencebased. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, selfreported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog comparison task. Our implementation is available at https://github.com/Weixin-Liang/ dialog_evaluation_CMADE.

HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

“…However, self-reported ratings suffer from bias and variance among different users . Denoising human ratings is still an open research problem (Liang et al, 2020e;.…”

“…A major bottleneck of these methods is that they require hand-labeling many dialog samples for individual datasets. Although Liang et al (2020e) denoise user self-reported ratings with the Shapley algorithm for dialog system evaluation, their method cannot be directly applied to dialogs without user ratings as in our setting. Our work is focusing on the problem that it is expensive and difficult to obtain user ratings.…”

“…Learning from weak supervision reduces annotation costs by utilizing noisy but cost-efficient labels (Ratner et al, 2020(Ratner et al, , 2016Liang et al, 2020e). One of the most popular forms of weak supervision is distant supervision, in which the records of an external knowledge base are heuristically aligned with data points to produce noisy labels for relationship extraction tasks (Bunescu and Mooney, 2007;Mintz et al, 2009;Hancock et al, 2018).…”

“…(2) Currently, self-reported user ratings are widely used to evaluate open-domain dialogs. However, self-reported ratings suffer from bias and variance among different users (Liang et al, 2020e). Although we could tell which dialog system is better by running statistical tests on a large number of noisy ratings, it is challenging to locate dialogs with bad performance reliably.…”

What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts

“…Sandbank et al (2018) present an approach for classifying low-quality conversations in commercial conversational assistants. Liang et al (2020) argued against the feasability of conversation-level quality prediction on a Likertscale and present a pairwise comparison model instead using methods that compensated for the high noise in user scores. Choi et al (2019) presents methods for both predicting user satisfaction and detecting conversation breakdowns at the turn level.…”