Caller Experience: A method for evaluating dialog systems and its automatic prediction

Abstract

In this paper we introduce a subjective metric for evaluating the performance of spoken dialog systems, Caller Experience (CE). CE is a useful metric for tracking the overall performance of a system in deployment, as well as for isolating individual problematic calls in which the system under-performs. The proposed CE metric differs from most performance evaluation metrics proposed in the past in that it is a) a subjective, qualitative rating of the call, and b) provided by expert, external listeners, not the callers themselves. The results of an experiment in which a set of human experts listened to the same calls three times are presented. The fact that these results show a high level of agreement among different listeners, despite the subjective nature of the task, demonstrates the validity of using CE as a standard metric. Finally, an automated rating system using objective measures is shown to perform at the same high level as the humans. This is an important advance, since it provides a way to reduce the human labor costs associated with producing a reliable CE.

Predicting user evaluations of spoken dialog systems using semi-supervised learning

“…Möller and Ward [9] further proposed a tripartite framework to evaluate SDS: "One part models the behavior of user and system during the interaction, the second one the perception and judgment processes taking place inside the user, and the third part models what matters to system designers and service providers." Similar to our work, Evanini et al [3] used a decision tree to predict caller experience, and Engelbrech et al [2] use Hidden Markov Models to predict the user judgements. However, the main differences from our work are: (1) we utilize crowdsourcing rather than experts [3] or true users [2] to get more annotated dialogs; (2) we leverage SSL to predict user evaluation, which is more suitable when the amount of labeled data is small.…”

“…Similar to our work, Evanini et al [3] used a decision tree to predict caller experience, and Engelbrech et al [2] use Hidden Markov Models to predict the user judgements. However, the main differences from our work are: (1) we utilize crowdsourcing rather than experts [3] or true users [2] to get more annotated dialogs; (2) we leverage SSL to predict user evaluation, which is more suitable when the amount of labeled data is small.…”

Modeling the user state for context-aware spoken interaction in ambient assisted living

“…Systems can also adapt to the user environment, as in the case of Ambient Intelligence applications [25,169]. A more sophisticated approach is to adapt the system to the user specifi knowledge and expertise, in which case the main research topics are the adaptation of systems to proficien y in the interaction language [112], age [161], different user expertise levels [49], and special needs [96]. Despite their complexity, these characteristics are to some extent rather static.…”

Ordinal regression for interaction quality prediction

“…The first kind asks the user to give a numerical rating and/or fill in a questionnaire about the dialogue after its end [2,3,4]. The second one involves experts instead of users [5,6]. In both cases, ratings are meant for system performance tracking and if necessary, system behaviour adaptation.…”