Interactive Robot Learning for Multimodal Emotion Recognition

Abstract

Interaction plays a critical role in skills learning for natural communication. In human-robot interaction (HRI), robots can get feedback during the interaction to improve their social abilities. In this context, we propose an interactive robot learning framework using multimodal data from thermal facial images and human gait data for online emotion recognition. We also propose a new decision-level fusion method for the multimodal classification using Random Forest (RF) model. Our hybrid online emotion recognition model focuses on the detection of four human emotions (i.e., neutral, happiness, angry, and sadness). After conducting offline training and testing with the hybrid model, the accuracy of the online emotion recognition system is more than 10% lower than the offline one. In order to improve our system, the human verbal feedback is injected into the robot interactive learning. With the new online emotion recognition system, a 12.5% accuracy increase compared with the online system without interactive robot learning is obtained.

SRG3: Speech-driven Robot Gesture Generation with GAN

“…2) Quantitative Evaluation: We also estimate the generated pose using a Average Position Error (APE) [20] as shown in Eq. 21, where T is the time steps and is equal to 2 Due to Covid19 and the lockdown, we couldn't use the real robot to run the experiments 126; M is the number of testing samples and is equal to 960 (30 batches with batch size 32); y real (m,t) and y generated (m,t) are the the ground truth and prediction of joint position y of sample m at time step t, respectively.…”

Multimodal User Feedback During Adaptive Robot-Human Presentations

“…They have previously performed well on feedback analysis tasks, like in the recent work by Jain et al (2021), who recently successfully used random forests to identify multimodal feedback in clips of test participants, or in the work by Soldner et al (2019), who successfully used random forests to classify whether participants in a study were lying based on multimodal cues. Yu and Tapus (2019) used random forests to classify emotions based on the combined modalities of thermal vision and body pose, finding that the random forest model successfully combined the modalities to achieve better performance than on either modality in isolation.…”

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

“…An interesting approach was proposed in Yu and Tapus ( 2019 ) for multimodal emotion recognition from thermal facial images and gait analysis. Here, interactive robot learning (IRL) was proposed to take advantage of human feedback obtained by the robot during HRI.…”