A Comprehensive Assessment of Dialog Evaluation Metrics

Abstract

Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

“…The same story also happens for Flow score, a state-of-the-art metric in the DSTC9 dataset. This observation is consistent with study from previous work (Yeh et al, 2021).…”

“…The common practice to show the effectiveness of a dialogue evaluation metric is to calculate the Pearson, Spearman's, and Kendall correlation between human evaluation and the automatic evaluation (Mehri and Eskénazi, 2020;Yeh et al, 2021). Table 2 list the correlations between automatic metrics and human evaluation.…”

“…FlowEval Can Provide Complementary Information to Other Methods. Similar to Yeh et al (2021), we test different combinations of metrics by directly averaging one metric with the best metrics on the three datasets, which are BERTScore on Controllable Dialogue dataset, DynaEval_emp on FED dataset, and Flow score on DSTC9 dataset. The last 4 rows of Table 2 show that FlowEval can consistently push the current correlation ceiling to a new level the most, while many other combinations improve little or even hurt performance.…”

“…Recent works tackle this problem by leveraging more sophisticated architectures and harnessing the power of large models (Mehri and Eskénazi, 2020). Although these recent metrics claim to show some progress towards higher correlation with humans, the gap between automatic metrics and human evaluation is still noticeable (Yeh et al, 2021). Automatic open-domain dialogue evaluation is still an open question, and extensive efforts have been made to improve performance from different angles (Pang et al, 2020;Ghazarian et al, 2019;Mehri and Eskenazi, 2020;Phy et al, 2020).…”

Mental Health Assessment for the Chatbots

“…Many researches on chatbot assessment are usually concerned about the local and technical metrics (e.g. fluency, diversity, interesting, informative, etc) (Mehri and Eskénazi, 2020a;Yeh et al, 2021). Under these criteria, chatbots can provide useful, interesting, and informative responses in online interactions with humans.…”