Group Emotion Detection Based on Social Robot Perception

Abstract

Social robotics is an emerging area that is becoming present in social spaces, by introducing autonomous social robots. Social robots offer services, perform tasks, and interact with people in such social environments, demanding more efficient and complex Human–Robot Interaction (HRI) designs. A strategy to improve HRI is to provide robots with the capacity of detecting the emotions of the people around them to plan a trajectory, modify their behaviour, and generate an appropriate interaction with people based on the analysed information. However, in social environments in which it is common to find a group of persons, new approaches are needed in order to make robots able to recognise groups of people and the emotion of the groups, which can be also associated with a scene in which the group is participating. Some existing studies are focused on detecting group cohesion and the recognition of group emotions; nevertheless, these works do not focus on performing the recognition tasks from a robocentric perspective, considering the sensory capacity of robots. In this context, a system to recognise scenes in terms of groups of people, to then detect global (prevailing) emotions in a scene, is presented. The approach proposed to visualise and recognise emotions in typical HRI is based on the face size of people recognised by the robot during its navigation (face sizes decrease when the robot moves away from a group of people). On each frame of the video stream of the visual sensor, individual emotions are recognised based on the Visual Geometry Group (VGG) neural network pre-trained to recognise faces (VGGFace); then, to detect the emotion of the frame, individual emotions are aggregated with a fusion method, and consequently, to detect global (prevalent) emotion in the scene (group of people), the emotions of its constituent frames are also aggregated. Additionally, this work proposes a strategy to create datasets with images/videos in order to validate the estimation of emotions in scenes and personal emotions. Both datasets are generated in a simulated environment based on the Robot Operating System (ROS) from videos captured by robots through their sensory capabilities. Tests are performed in two simulated environments in ROS/Gazebo: a museum and a cafeteria. Results show that the accuracy in the detection of individual emotions is 99.79% and the detection of group emotion (scene emotion) in each frame is 90.84% and 89.78% in the cafeteria and the museum scenarios, respectively.

New Trends in Emotion Recognition Using Image Analysis by Neural Networks, a Systematic Review

“…There are also situations in which the proposed methods exceed the state-of-the-art performances [ 38 , 39 , 54 , 62 , 74 , 81 , 101 ]. Similarly, context-aware solutions for emotion recognition [ 47 , 49 , 50 , 98 ] or practical solutions [ 37 , 124 , 127 , 130 , 133 , 159 ] demonstrate promising results.…”

“…Regarding the computer vision domain, neural networks have been successfully used in image classification and more specifically, face identification and facial emotion recognition applications. Besides the main utility in surveillance systems, neural networks have also begun to be used in medical diagnosis applications (to identify patient conditions [ 69 , 94 , 95 ]) or in applications that involve interaction with a user [ 96 , 97 , 98 , 99 , 100 ].…”

The impact of loose‐parts‐play on schoolyard social participation of children with and without disabilities: A case study

“…This study used advanced sensing methodologies to gain new insights into playground social dynamics. New directions in research that have not yet been applied to the playground context, such as μEMA (Ponnada et al, 2022) or machine-learning emotion detection (Guerra et al, 2022), are promising for detecting fine-tuning differences in valence of social interactions and may contribute to future research on social participation and field interventions.…”

An E2ED-based Approach to Custom Robot Navigation and Localization

“…In today's social development, the implementation of mobile robots for problem-solving in industrial and non-industrial environments is of great relevance [3]. Applications in rural or agricultural areas, as well as those involving direct interaction with humans [4], [5], are of great current interest. In this respect, mobile robotic platforms have been proposed that can be used in a greenhouse to transport vegetables [6], [7], in some cases with safe interaction systems with humans.…”