Conversations with a virtual human: Synthetic emotions and human responses

Abstract

To test whether synthetic emotions expressed by a virtual human elicit positive or negative emotions in a human conversation partner and affect satisfaction towards the conversation, an experiment was conducted where the emotions of a virtual human were manipulated during both the listening and speaking phase of the dialogue. Twenty-four participants were recruited and were asked to have a real conversation with the virtual human on six different topics. For each topic the virtual human's emotions in the listening and speaking phase were different, including positive, neutral and negative emotions. The results support our hypotheses that (1) negative compared to positive synthetic emotions expressed by a virtual human can elicit a more negative emotional state in a human conversation partner, (2) synthetic emotions expressed in the speaking phase have more impact on a human conversation partner than emotions expressed in the listening phase, (3) humans with less speaking confidence also experience a conversation with a virtual human as less positive, and (4) random positive or negative emotions of a virtual human have a negative effect on the satisfaction with the conversation. These findings have practical implications for the treatment of social anxiety as they allow therapists to control the anxiety evoking stimuli, i.e. the expressed emotion of a virtual human in a virtual reality exposure environment of a simulated conversation. In addition, these findings may be useful to other virtual applications that include conversations with a virtual human.

Reporting Mental Health Symptoms: Breaking Down Barriers to Care with Virtual Human Interviewers

“…Interspersed appropriately during an interview, the virtual human interviewers use verbal and non-verbal backchannels (e.g., utterances of agreement such as "mhm" or head nods) to build rapport with the interviewee. Indeed, virtual human interviewers that employ such backchannels when appropriate to the conversation create greater feelings of rapport than virtual human interviewers that employ them at random during the interview (e.g., Gratch et al, 2013;Qu et al, 2014). As with Rapport Agents, when virtual human interviewers use these rapport-building behaviors in this way, they are able to prompt disclosure from interviewees (Gratch et al, 2013).…”

“…Some virtual humans can be used to interview people in a natural way (i.e., via conversational speech). Akin to the "Rapport Agents" described above, these virtual human interviewers have been designed to build rapport with users specifically during interviews (e.g., Gratch et al, 2013;Qu et al, 2014), including clinical interviews (e.g., Bickmore et al, 2005;DeVault et al, 2014;Lucas et al, 2014;Rizzo et al, 2016). Interspersed appropriately during an interview, the virtual human interviewers use verbal and non-verbal backchannels (e.g., utterances of agreement such as "mhm" or head nods) to build rapport with the interviewee.…”

Exploring the affective, motivational and cognitive effects of pedagogical agent enthusiasm in a multimedia learning environment

“…The social agency theory is supported by findings of a recent meta-analysis; it was revealed that learners achieved more from multimedia learning system with a pedagogical agent than a system without an agent (see for a review [3]). Advances in multimedia have allowed pedagogical agents to express synthetic emotions and nonverbal behaviours [4]. Hence, these advances open up a myriad of potential social learning benefits, due to the simulated socio-emotive relationships between learners and computer agents [5,6].…”

Review of spoken dialogue systems

“…Some examples of such behaviour are to avoid user negative emotions due to system errors , to favour engagement by diminishing boredom (Baker, D'Mello, Rodrigo, & Graesser, 2010), to maximize satisfaction (Lebai Lufti, Fernández-Martínez, Lucas-Cuesta, López-Lebón, & Montero, 2013), or to foster positive emotions to adhere to healthy habits (Creed & Beale, 2012). However, in some application domains it might also be useful to render or provoke negative states, for example, for emotional mirroring, or to try to stress the users for a specific purpose, for example, for the treatment of different types of anxiety (Callejas, Ravenet, Ochs, & Pelachaud;2014;Qu, Brinkman, Ling, Wiggers, & Heynderickx, 2014).…”