Problem detection in human–machine interactions based on facial expressions of users

Abstract

This paper describes research into audiovisual cues to communication problems in interactions between users and a spoken dialogue system. The study consists of two parts. First, we describe a series of three perception experiments in which subjects are offered film fragments (without any dialogue context) of speakers interacting with a spoken dialogue system. In half of these fragments, the speaker is or becomes aware of a communication problem. Subjects have to determine by forced choice which are the problematic fragments. In all three tests, subjects are capable of performing this task to some extent, but with varying levels of correct classifications. Second, we report results of an observational analysis in which we first attempt to relate the perceptual results to visual features of the stimuli presented to subjects, and second to find out which visual features actually are potential cues for error detection. Our major finding is that more problematic contexts lead to more dynamic facial expressions, in line with earlier claims that communication errors lead to marked speaker behaviour. We conclude that visual information from a userÕs face is potentially beneficial for problem detection.

Feedback interpretation based on facial expressions in human-robot interaction

“…Fortunately, it seems likely that people often will show some striking facial expression in the presence of a communication problem. This assumption is confirmed by the experiments of Barkhuysen et al [5] and also by preliminary studies we carried out.…”

“…The recognition performance of the subjects can serve as a baseline for the development of automatic recognition approaches. We compare our results with a study of Barkhuysen et al [5]. The authors showed video fragments of people interacting with a spoken dialog system to subjects, who had to decide whether there was a communication problem in a particular interaction.…”

“…Barkhuysen et al [5] conducted three experiments where subjects watched film fragments of speakers interacting with an oral train timetable dialog system. The subjects decided whether or not there was a communication problem present in the shown situation.…”

Culture and Facial Expressions: A Case Study with a Speech Interface

“…Perception study is a widely used method for evaluating facial expressiveness, in terms of type and/or intensity of expression [3], [9], [10]. It involves showing a series of videos or pictures to the participants and asking them what kind of expression is perceived by the viewer and/or rating how intense an expression is.…”

“…a facial expression in a neutral, non-erroneous context, was included in the perception test. The neutral videos were selected from dialogue fragments without errors, and without clearly perceivable muscle movements within a timeframe of 5 seconds, following best practice procedures outlines in related studies [3]. Neutral videos were used later as a baseline in terms of expressiveness.…”