See You See Me

Abstract

We focus on a fundamental looking behavior in human-robot interactions – gazing at each other’s face. Eye contact and mutual gaze between two social partners are critical in smooth human-human interactions. Therefore, investigating at what moments and in what ways a robot should look at a human user’s face as a response to the human’s gaze behavior is an important topic. Toward this goal, we developed a gaze-contingent human-robot interaction system, which relied on momentary gaze behaviors from a human user to control an interacting robot in real time. Using this system, we conducted an experiment in which human participants interacted with the robot in a joint attention task. In the experiment, we systematically manipulated the robot’s gaze toward the human partner’s face in real time and then analyzed the human’s gaze behavior as a response to the robot’s gaze behavior. We found that more face looks from the robot led to more look-backs (to the robot’s face) from human participants and consequently created more mutual gaze and eye contact between the two. Moreover, participants demonstrated more coordinated and synchronized multimodal behaviors between speech and gaze when more eye contact was successfully established and maintained.

Automatic Visual Attention Detection for Mobile Eye Tracking Using Pre-Trained Computer Vision Models and Human Gaze

“…Prasov and Chai [ 31 ] developed a system that combines speech and eye gaze to enhance reference resolution in conversational interfaces. Xu et al [ 32 ] investigated the role of mutual gaze in a human-robot collaboration setting and found that maintaining eye contact leads to improved multimodal interaction behavior of users, i.e., more synchronized and coordinated. Baur et al [ 33 ] implemented NovA, a system for analyzing and interpreting social signals in multi-modal interactions with a conversational agent, which integrates eye tracking technology.…”

Issues in the Development of Conversation Dialog for Humanoid Nursing Partner Robots in Long-Term Care

“…The issue of conversation with Pepper includes expressions such as robot gaze [34], eye blink synchrony [35], eye contact [36], and speech [37]. This issue between older persons and Pepper with a conversation function in the application named "Kenkou-oukoku TALK for Pepper [38]" is vocalization with less intonation.…”

Reading the Mind in Robots: How Theory of Mind Ability Alters Mental State Attributions During Human-Robot Interactions

“…The behaviors of agents, both human and robot, were found to affect each other in real-time. Switching robot attention between a human partner's face and object in a joint-attention task, fostered human-robot coordination (Xu et al, 2016). Others have similarly studied how ToM attributions of robots will influence HRI.…”

“…When the robot had human-like characteristics, gaze cues guided human responses, suggesting that intentions were being inferred, thus improving HRI. Research has also examined settings where humans and robots participated in joint-attention tasks (Xu, Zhang, & Yu, 2016). The behaviors of agents, both human and robot, were found to affect each other in real-time.…”