Facial Emotion Recognition from an Unmanned Flying Social Robot for Home Care of Dependent People

Abstract

This work is part of an ongoing research project to develop an unmanned flying social robot to monitor dependants at home in order to detect the person’s state and bring the necessary assistance. In this sense, this paper focuses on the description of a virtual reality (VR) simulation platform for the monitoring process of an avatar in a virtual home by a rotatory-wing autonomous unmanned aerial vehicle (UAV). This platform is based on a distributed architecture composed of three modules communicated through the message queue telemetry transport (MQTT) protocol: the UAV Simulator implemented in MATLAB/Simulink, the VR Visualiser developed in Unity, and the new emotion recognition (ER) system developed in Python. Using a face detection algorithm and a convolutional neural network (CNN), the ER System is able to detect the person’s face in the image captured by the UAV’s on-board camera and classify the emotion among seven possible ones (surprise; fear; happiness; sadness; disgust; anger; or neutral expression). The experimental results demonstrate the correct integration of this new computer vision module within the VR platform, as well as the good performance of the designed CNN, with around 85% in the F1-score, a mean of the precision and recall of the model. The developed emotion detection system can be used in the future implementation of the assistance UAV that monitors dependent people in a real environment, since the methodology used is valid for images of real people.

Application of Social Robots in Healthcare: Review on Characteristics, Requirements, Technical Solutions

“…This latter robotic field in particular is rapidly progressing and benefits other relevant technologies [ 13 , 14 , 15 ]. A classification of different robots according to their embodiments is listed in Table 3 .…”

Video Processing from a Virtual Unmanned Aerial Vehicle: Comparing Two Approaches to Using OpenCV in Unity

“…From the experiments conducted, it can be concluded that interoperability between Unity and other languages is improved in terms of data flow by using a subscribe/publish communication protocol such as MQTT rather than creating a native plugin as this is dependent on running on the same thread as Unity. MQTT had already been used by our research team in recent applications related to simulated UAV indoors (e.g., [9,39,41]). All video processing was done externally with MQTT, allowing more FPS to be processed.…”

“…MQTT has been selected in our approach as it has already shown excellent results in our previous works for MATLAB/Simulink integration [9,39]. Moreover, the combination of an external module programmed in Python for image processing and communicated via MQTT has recently been implemented to evaluate emotion recognition as part of ongoing research [40,41].…”

PlatypOUs—A Mobile Robot Platform and Demonstration Tool Supporting STEM Education

“…The link between the headset and the robotic platform was created using the Message Queueing Telemetry Transport (MQTT) communication protocol [ 25 ]. MQTT is a lightweight protocol for IoT messaging and is widely used in robot control systems [ 26 , 27 , 28 , 29 , 30 , 31 ]. It was our choice because it has a small computational overhead and was easy to integrate into our system.…”