Untitled

Comparing objective and subjective measures of usability in a human-robot dialogue system

“…While the predictor functions give useful insights into the relative contribution of the objective measures to the subjective user satisfaction, the R 2 values are generally lower than those found in other PARADISE-style evaluations. For example, Walker et al (1998) reported an R 2 value of 0.38, the values reported by on the training sets ranged from 0.39 to 0.56, Litman and Pan (2002) reported an R 2 value of 0.71, while the R 2 values reported by Möller et al (2008) for linear regression models similar to those presented here were between 0.22 and 0.57. The low R 2 values from this analysis clearly suggest that, while the factors included in Table 6 did affect users' opinions-particularly their opinion of the robot as a partner and the change in their reported emotional state-the users' subjective judgements were also affected by factors other than those captured by the objective measures considered here.…”

“…We propose two possible explanations for this difference. First, the system analysed by Litman and Pan (2002) was an information-seeking dialogue system, in which efficient access to the information is an important criterion. The current system, on the other hand, has the goal of joint task execution, and pure efficiency is a less compelling measure of dialogue quality in this setting.…”

“…The PARADISE framework (PARAdigm for DIalogue System Evaluation; Walker et al (1997)) describes a method for using data to derive a performance function that predicts user-satisfaction scores from the results on other, more easily computed measures. PARADISE uses stepwise multiple linear regression to model user satisfaction based on measures representing the performance dimensions of task success, dialogue quality, and dialogue efficiency, and has been applied to a wide range of systems (e.g., Litman and Pan, 2002;Möller et al, 2008). If the resulting performance function can be shown to predict user satisfaction as a function of other, more easily measured system properties, it will be widely applicable: in addition to making it possible to evaluate systems based on automatically available data from log files without the need for extensive experiments with users, for example, such a performance function can be used in an online, incremental manner to adapt system behaviour to avoid entering a state that is likely to reduce user satisfaction, or can be used as a reward function in a reinforcement-learning scenario (Walker, 2000).…”

The State of Speech in HCI: Trends, Themes and Challenges

“…Wilkie et al (2007) demonstrated that providing hidden menu options versus explicit options to encourage proactive user requests may prevent users from successfully completing phone-based banking tasks. Litman & Pan (2002) evaluated an online train schedule retrieval system accessed via telephone, revealing performance improvements when the system predicts ASR problems and adapts to more conservative dialogue strategies. In another paper, a field study compared call routing in a cell center environment (Suhm et al, 2002).…”

How affordances of chatbots cross the chasm between social and traditional enterprise systems

“…It is already fifty years ago, since the well-known program ELIZA replied to human input based on keyword recognition (Weizenbaum 1966). Aside from the introduced concepts, a plethora of related concepts have emerged over time, e.g., conversational interface (Knight 2016), chat agent (Crutzen et al 2011), chatterbot (Mauldin 1994), and dialogue system (Litman and Pan 2002). However, "whether you call these things digital assistants, conversational interfaces or just chatbots, the basic concept is the same: achieve some result by conversing with a machine in a dialogic fashion, using natural language" (Dale 2016, p. 811).…”