Emotion Detection for Social Robots Based on NLP Transformers and an Emotion Ontology

Abstract

For social robots, knowledge regarding human emotional states is an essential part of adapting their behavior or associating emotions to other entities. Robots gather the information from which emotion detection is processed via different media, such as text, speech, images, or videos. The multimedia content is then properly processed to recognize emotions/sentiments, for example, by analyzing faces and postures in images/videos based on machine learning techniques or by converting speech into text to perform emotion detection with natural language processing (NLP) techniques. Keeping this information in semantic repositories offers a wide range of possibilities for implementing smart applications. We propose a framework to allow social robots to detect emotions and to store this information in a semantic repository, based on EMONTO (an EMotion ONTOlogy), and in the first figure or table caption. Please define if appropriate. an ontology to represent emotions. As a proof-of-concept, we develop a first version of this framework focused on emotion detection in text, which can be obtained directly as text or by converting speech to text. We tested the implementation with a case study of tour-guide robots for museums that rely on a speech-to-text converter based on the Google Application Programming Interface (API) and a Python library, a neural network to label the emotions in texts based on NLP transformers, and EMONTO integrated with an ontology for museums; thus, it is possible to register the emotions that artworks produce in visitors. We evaluate the classification model, obtaining equivalent results compared with a state-of-the-art transformer-based model and with a clear roadmap for improvement.

Estimation of Organizational Competitiveness by a Hybrid of One-Dimensional Convolutional Neural Networks and Self-Organizing Maps Using Physiological Signals for Emotional Analysis of Employees

“…According to Li et al [ 5 ], a brain–computer interface-based emotion recognition scheme with an improved particle swarm optimization for feature selection was employed with an accuracy of 95%. Graterol et al [ 60 ] proposed a method for emotion recognition and achieved an accuracy of around 53% for classification tasks. Seo et al [ 21 ] used machine learning methods for boredom classification using EEG and GSR data to reach an accuracy of 89%.…”

Ontological Model for Contextual Data Defining Time Series for Emotion Recognition and Analysis

“…EMONTO [29] is an extensible ontology that represents emotions under different categorization proposals. The key class is Emotion, which has a category (hasCategory) ac-cording to a Category class.…”

“…Some of them are used to model emotions for the task of emotion recognition from text [3,7,70,71,80,81]. Others model emotions for human-computer interactions [21,22,51] or human-robot interactions [29]. Sometimes, ontological models in Affective Computing are applied to obtain a specific goal such as detecting phobia/philia [5], or standardizing the main emotion models and mapping together different representations Zhang et al [86] introduced BIO_EMOTION, an ontologybased context model for emotion recognition which allows a modeling of user contexts, including user profile, EEG data, the situation and environment factors, as well as supports reasoning on the user's emotional state(s).…”

A Multi-modal Visual Emotion Recognition Method to Instantiate an Ontology

“…Since the analysis of human emotions is generally performed to associate emotions to certain events or objects and consequently to make decisions or for further analysis, keeping this information in semantic repositories offers a wide range of possibilities for implementing smart applications (Bertola and Patti, 2016;Chen et al, 2016;Cavaliere and Senatore, 2019;Graterol et al, 2021).…”

“…To support decision making and post-analysis from the analysis of emotions, we use an emotion ontology, called EMONTO (Graterol et al, 2021). EMONTO is an extensible emotion ontology, that can be integrated with other specific domain ontologies.…”