Evaluating discourse understanding in spoken dialogue systems

Abstract

This paper describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. Discourse understanding means utterance understanding taking the context into account. Since the measure needs to be determined based on its correlation with the system's performance, conventional measures, such as the concept error rate, cannot be easily applied. Using the multiple linear regression analysis, we have previously shown that the weighted sum of various metrics concerning dialogue states can be used for the evaluation of discourse understanding in a single domain. This paper reports the progress of our work: verification of our approach by additional experiments in another domain. The support vector regression method performs better than the multiple linear regression method in creating the measure, indicating non-linearity in mapping the metrics to the system's performance. The results give strong support for our approach and hint at its suitability as a universal evaluation measure for discourse understanding.

Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems

“…Note that, although the values of correlation ratio seem rather low, the correlation often becomes low when it comes to subjective judgments (Higashinaka et al, 2004). Considering that we deal with chat-oriented dialogues, which are less restricted than task-oriented ones, we consider the current values of correlation ratio to be acceptable.…”

The Dialog State Tracking Challenge Series: A Review

“…Two metrics, Update precision and Update accuracy measure the accuracy and precision of updates to the top scoring hypothesis from one turn to the next. For more details, see Higashinaka et al (2004), which finds these metrics to be highly correlated with dialog success in their data.…”

Ranking Multiple Dialogue States by Corpus Statistics to Improve Discourse Understanding in Spoken Dialogue Systems

“…1 Architecture of a spoken dialogue system. This figure is a modified version of the diagram we used in [11].…”

“…Although Higashinaka et al [11] proposed creating an evaluation measure for discourse understanding by finding a measure that correlates closely with the performance of a dialogue system, the measure assumes that the system holds a single dialogue state. The best measure they propose is based on the precision of the update of a dialogue state (called update precision), which is difficult to calculate when a system has multiple dialogue states because it is not clear whether the sequence of dialogue states with a different understanding history can be used to calculate the update precision.…”