Comparing objective and subjective measures of usability in a human-robot dialogue system

Abstract

We present a human-robot dialogue system that enables a robot to work together with a human user to build wooden construction toys. We then describe a study in which naïve subjects interacted with this system under a range of conditions and then completed a user-satisfaction questionnaire. The results of this study provide a wide range of subjective and objective measures of the quality of the interactions. To assess which aspects of the interaction had the greatest impact on the users' opinions of the system, we used a method based on the PARADISE evaluation framework (Walker et al., 1997) to derive a performance function from our data. The major contributors to user satisfaction were the number of repetition requests (which had a negative effect on satisfaction), the dialogue length, and the users' recall of the system instructions (both of which contributed positively).

Using Embodied Multimodal Fusion to Perform Supportive and Instructive Robot Roles in Human-Robot Interaction

“…The R 2 values of this study are in the same range as the values of our previous user evaluations [11,14]. However, the values are not as high as those reported in [33] and [20].…”

Multivariate evaluation of interactive robot systems

“…Prior research has followed an evaluation paradigm called PARADISE (PARAdigm for DIalogue System Evaluation)-a regression-based approach to evaluating the performance of dialog systems (Walker et al 1997). In a human-robot collaboration task, Foster et al (2009) used measurements of user behaviors, such as the number of repeated inquiries, to assess users' perceptions of the robot system. Similarly, Peltason et al (2012) investigated the extent to which measurements, such as the duration of interaction and user actions, in an objectlearning task predict users' perceptions of their interactions with a humanlike robot.…”

“…This approach builds on multiple regression to model the relationship between a set of design variables and an interaction outcome of interest as a linear system in order to determine which design variables predict the interaction outcome and to what extent these predictors affect it. Our proposed approach is different from prior uses of regressionbased analysis in studying interactions with interactive robot systems, which focus on how emergent aspects of the interaction between the user and the interactive robot system, such as the number of repeated inquiries by the user (Foster et al 2009) or the total duration of the interaction (Peltason et al 2012), might predict user experience with the robot system. In contrast, our proposed method involves directly manipulating each design variable to vary across a continuous range and assessing its relative contribution to predicting the outcome variable.…”

“…Advances in robotic technology are enabling the development of robot systems that interact with humans in a wide range of applications, including education (Tanaka et al 2007;Szafir and Mutlu 2012), storytelling (Mutlu et al 2006;Ham et al 2011), museum guidance (Huang et al 2014), coaching in weight loss and exercise (Kidd 2008;Fasola and Mataric 2012), and collaborative work (Foster et al 2009;Huang and Mutlu 2012;Salem et al 2012). In such interactions, robot systems employ a wide range of behaviors including elements of spoken and nonverbal language to improve how effectively they support these applications and their users' experiences.…”

Evaluating embodied conversational agents in multimodal interfaces

“…dialogue duration, system and user turn duration (Gibbon et al 2000), system response delay (Price et al 1992), can be transferred to multimodal interaction unmodified -system response delay, for example, has been used before as a parameter for multimodal systems to measure dialogue efficiency (Foster et al 2009). Their measurement is based on the definition of user and system turns as described above.…”