Survey on AI-Based Multimodal Methods for Emotion Detection

Maréchal, Catherine; Mikołajewski, Dariusz; Tyburek, Krzysztof; Prokopowicz, Piotr; Bougueroua, Lamine; Ancourt, Corinne; Węgrzyn-Wolska, Katarzyna

doi:10.1007/978-3-030-16272-6_11

Cited by 87 publications

(42 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Detailed information on the current state-of-the-art in a more generalized perspective, we refer the reader to the surveys [ 2 , 11 , 43 , 44 , 45 , 46 , 47 ] and references therein, where a comprehensive review of the latest work on ER using ML and physiological signals can be found, highlighting the main achievements, challenges, take-home messages, and possible future opportunities.…”

Section: State Of the Artmentioning

confidence: 99%

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Bota

Wang²,

Fred

et al. 2020

Sensors

View full text Add to dashboard Cite

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.

show abstract

Section: State Of the Artmentioning

confidence: 99%

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Bota

Wang²,

Fred

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…The robot has to understand the explicit and implicit communicative cues people produce with their body, mostly the affective expressions. Although many systems have been proposed to detect emotion from facial expressions [59,60], they often have relatively low accuracy, leading to too quick shift in interpretation and quite demanding requirements both about face resolution and the expression of facial cues [61], often required to be unnaturally expanded to be recognized. Work has still to be done on body expressions [62] and on subtle cues, whose detection is also limited by sensor resolution, the learning models which cannot come with too high complexity, and the situations where interaction actually occurs, with subjects moving fast in front of the robot, and reaching positions out of the camera range.…”

Section: Visual Interaction: Human To Robotmentioning

confidence: 99%

Communication in Human-Robot Interaction

Bonarini

2020

Curr Robot Rep

View full text Add to dashboard Cite

Purpose of Review To present the multi-faceted aspects of communication between robot and humans (HRI), putting in evidence that it is not limited to language-based interaction, but it includes all aspects that are relevant in communication among physical beings, exploiting all the available sensor channels. Recent Findings For specific purposes, machine learning algorithms could be exploited when data sets and appropriate algorithms are available. Summary Together with linguistic aspects, physical aspects play an important role in HRI and make the difference with respect to the more limited human-computer interaction (HCI). A review of the recent literature about the exploitation of different interaction channels is presented. The interpretation of signals and the production of appropriate communication actions require to consider psychological, sociological, and practical aspects, which may affect the performance. Communication is just one of the functionalities of an interactive robot and, as all the others, will need to be benchmarked to support the possibility for social robots to reach a real market.

show abstract

“…Moreover, images of a person are as easy to record as their speech, since a camera record both image and sound whereas physiological measures are clearly less convenient to be recorded. The accuracy of expression recognition is usually improved when it combines the analysis of human expressions from multimodal forms all together [14]. However, considering only a sub-part of channels such as only the face of a person or the posture stays an important way to improve knowledge of the whole domain since multi-modal approaches generally combine specific approaches dedicated to a single channel [15].…”

Section: Related Workmentioning

confidence: 99%