Achieving rapport with turn-by-turn, user-responsive emotional coloring

Abstract

People in dialog use a rich set of nonverbal behaviors, including variations in the prosody of their utterances. Such behaviors, often emotion-related, call for appropriate responses, but today's spoken dialog systems lack the ability to do this. Recent work has shown how to recognize user emotions from prosody and how to express system-side emotions with prosody, but demonstrations of how to combine these functions to improve the user experience have been lacking. Working with a corpus of conversations with students about graduate school, we analyzed the emotional states of the interlocutors, utterance by utterance, using three dimensions: activation, evaluation, and power. We found that the emotional coloring of the speaker's utterance could be largely predicted from the emotion shown by her interlocutor in the immediately previous utterance. This finding enabled us to build Gracie, the first spoken dialog system that recognizes a user's emotional state from his or her speech and gives a response with appropriate emotional coloring. Evaluation with 36 subjects showed that they felt significantly more rapport with Gracie than with either of two controls. This shows that dialog systems can tap into this important level of interpersonal interaction using today's technology.

I Feel You: The Design and Evaluation of a Domotic Affect-Sensitive Spoken Conversational Agent

“…A similar finding was found in [38]. The NEMOHIFI version's response (NRG content) was considerably lengthier, and the addition of prosody has made it even more lengthy than in previous experiments.…”

Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines

“…They quantitatively measure the rhythm entrainment between speakers as the latency of the first pitch accent after a turn exchange divided by the rate of pitch accents in the utterance preceding the turn exchange. Finally, [78] proposed using a machine learning algorithm to predict the emotional coloring (valence, activation, power) of an utterance based on the emotional coloring of the previous utterance.…”

Towards Interactive Agents that Infer Emotions from Voice and Context Information

“…For example, in (Bevacqua et al, 2010) vocal cues are used to generate backchannels (i.e., non-intrusive signals provided during the speaker's turn). Acosta and Ward proposed a system that uses speech and prosody variation to build rapport between human and agent (Acosta and Ward, 2011), and Cavazza et al used vocal signals for character-based interactive storytelling (Cavazza et al, 2009). Furthermore, the virtual human SimSensei Kiosk uses voice, speech and other features to analyse user emotions in the context of healthcare decision support (DeVault et al, 2014).…”