Building an Emotionally Responsive Avatar with Dynamic Facial Expressions in Human—Computer Interactions

Abstract

The role of affect has been long studied in human–computer interactions. Unlike previous studies that focused on seven basic emotions, an avatar named Diana was introduced who expresses a higher level of emotional intelligence. To adapt to the users various affects during interaction, Diana simulates emotions with dynamic facial expressions. When two people collaborated to build blocks, their affects were recognized and labeled using the Affdex SDK and a descriptive analysis was provided. When participants turned to collaborate with Diana, their subjective responses were collected and the length of completion was recorded. Three modes of Diana were involved: a flat-faced Diana, a Diana that used mimicry facial expressions, and a Diana that used emotionally responsive facial expressions. Twenty-one responses were collected through a five-point Likert scale questionnaire and the NASA TLX. Results from questionnaires were not statistically different. However, the emotionally responsive Diana obtained more positive responses, and people spent the longest time with the mimicry Diana. In post-study comments, most participants perceived facial expressions on Diana’s face as natural, four mentioned uncomfortable feelings caused by the Uncanny Valley effect.

Emotion Detection from Speech: A Comprehensive Approach Using Speech-to-Text Transcription and Ensemble Learning

“…This work contributes to the growing field of affective computing, where the goal is to imbue computers and digital interfaces with the ability to recognize and respond to human emotions. The advancements presented in this paper have implications for the design of emotionally engaging avatars in applications ranging from virtual assistants to gaming [4].…”

The emotions effect on a virtual characters design–A student perspective analysis

“…Given numerous human-robot interaction (HRI)-related studies (Jack et al, 2014;Wittig et al, 2015;Mcginn, 2020;Wang et al, 2021), where emotions are managed using Ekman's Facial Action Coding System (FACS) (Ekman and Friesen, 1978), our research focuses on 6 basic emotions: happiness, surprise, disgust, sadness, anger, and fear (Ekman et al, 1969;Piórkowska and Wrobel, 2017).…”

“…In a study by Chao et al (2012), the recognition of emotions by the user is included but the communication of emotions by the avatar remains unattended. A comparative analysis by Wang et al (2021) found a greater user receptiveness when using an avatar with emotional responses unlike using an avatar with a flat facial expression.…”

Avatar Reaction to Multimodal Human Behavior

“…However, the challenge of building an interactive virtual agent differs from that of building an interactive robot. Heting et al [14] discuss the use of virtual agents, specifically avatars, in human-computer interactions. The paper presents an emotionally responsive avatar named Diana, which recognizes human affect and responds with natural facial expressions to improve the user experience in the interaction.…”