Facial Emotion Recognition from an Unmanned Flying Social Robot for Home Care of Dependent People

Abstract

This work is part of an ongoing research project to develop an unmanned flying social robot to monitor dependants at home in order to detect the person’s state and bring the necessary assistance. In this sense, this paper focuses on the description of a virtual reality (VR) simulation platform for the monitoring process of an avatar in a virtual home by a rotatory-wing autonomous unmanned aerial vehicle (UAV). This platform is based on a distributed architecture composed of three modules communicated through the Message Queue Telemetry Transport (MQTT) protocol: the UAV Simulator implemented in MATLAB/Simulink, the VR Visualiser developed in Unity, and the new emotion recognition (ER) System developed in Python. Using a face detection algorithm and a convolutional neural network (CNN), the ER System is able to detect the person’s face in the image captured by the UAV’s on-board camera and classify the emotion among seven possible ones (surprise, fear, happiness, sadness, disgust, anger or neutral expression). The experimental results demonstrate the correct integration of this new computer vision module within the VR platform, as well as the good performance of the designed CNN, with around 85% in the F1-score, a mean of the precision and recall of the model. The developed emotion detection system can be used in the future implementation of the assistance UAV that monitors dependent people in a real environment, since the methodology used is valid for images of real people.

Multimodal Emotional Understanding in Robotics

“…Other experimentation could be a sophisticated simulation [9]. In this experiment, worse and weird cases can be tested without affecting real persons; however, the results will be conditioned without much rigor.…”

“…They argue that the robot can perform a good emotion recognition process in real-time by adding an extra device. Unlike Greco et al [8], Martinez et al [9] use an external server instead of the extra device. The models used might be more complex, and their updating or deployment would require more effort.…”

“…The models used might be more complex, and their updating or deployment would require more effort. Martinez et al [9] use drones for capturing the inputs but do not send feedback but analyze and adapt future responses to different situations. For emotional recognition, they use a convolutional neural network.…”