Real-time emotion generation in human-robot dialogue using large language models

Abstract

Affective behaviors enable social robots to not only establish better connections with humans but also serve as a tool for the robots to express their internal states. It has been well established that emotions are important to signal understanding in Human-Robot Interaction (HRI). This work aims to harness the power of Large Language Models (LLM) and proposes an approach to control the affective behavior of robots. By interpreting emotion appraisal as an Emotion Recognition in Conversation (ERC) tasks, we used GPT-3.5 to predict the emotion of a robot’s turn in real-time, using the dialogue history of the ongoing conversation. The robot signaled the predicted emotion using facial expressions. The model was evaluated in a within-subjects user study (N = 47) where the model-driven emotion generation was compared against conditions where the robot did not display any emotions and where it displayed incongruent emotions. The participants interacted with the robot by playing a card sorting game that was specifically designed to evoke emotions. The results indicated that the emotions were reliably generated by the LLM and the participants were able to perceive the robot’s emotions. It was found that the robot expressing congruent model-driven facial emotion expressions were perceived to be significantly more human-like, emotionally appropriate, and elicit a more positive impression. Participants also scored significantly better in the card sorting game when the robot displayed congruent facial expressions. From a technical perspective, the study shows that LLMs can be used to control the affective behavior of robots reliably in real-time. Additionally, our results could be used in devising novel human-robot interactions, making robots more effective in roles where emotional interaction is important, such as therapy, companionship, or customer service.

Does a Robot’s Gaze Behavior Affect Entrainment in HRI?

“…Few researchers have started addressing this gap. For example, in a recent study [36] examined the relationship between robot and human gaze behavior. The study involved a within-subjects design where 33 participants interacted with a Furhat robot in two experimental conditions: Fixed Gaze and Gaze Aversion.…”

“…If robots do not exhibit gaze aversions, then users may have to put in extra effort to avoid frequent mutual gaze with the robot, which can make the interaction more difficult. In subsequent work, [35] utilized data collected in [36] and explored the relationship between gaze and speech entrainment. PRAAT toolkit was employed to extract mean pitch values of the participants' and robots' speech at each turn exchange.…”

“…It suggests that the closer speakers get, the lesser the social distance between them [3]. In the context of HRI, research has shown that people tend to avert their gaze less when a robot exhibits well-timed gaze aversion behavior [36]. It indicates that human-like gaze aversion behavior in robots can have a positive influence on overall experience and ease of communicating with the robot.…”

“…Therefore, we believe that employing comprehensive features and extensive analysis could shed additional light on the relationship between speech entrainment and gaze behavior. Hence, in a similar experimental setup, as the one used in [36], we hypothesize that participants will entrain more with the robot when it exhibits well-timed gaze aversions during an interaction (H1).…”

“…However, the relationship between gaze and entertainment in HMI has received less attention. In a recent study conducted by [35], speech entrainment was analyzed by measuring the mean pitch of speech collected from 33 participants subjected to two modes of robot's gaze behavior (fixed vs. variable) described in [36]. However, the results indicated no significant differences between the two conditions.…”