Multimodal emotion recognition from expressive faces, body gestures and speech

Caridakis, George; Castellano, Ginevra; Kessous, Loïc; Raouzaiou, Amaryllis; Malatesta, Lori; Asteriadis, Stylianos; Karpouzis, Kostas

doi:10.1007/978-0-387-74161-1_41

Cited by 112 publications

(56 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Their study shows that sadness and anger are detected more easily from speech, while the recognition of joy and fear is less reliable. Caridakis et al [34] obtained 93.30% and 76.67% accuracy to identify anger and sadness, respectively, from speech, using 377 features based on intensity, pitch, Mel-Scale Frequency Cepstral Coefficients (MFCC), Bark spectral bands, voiced segment characteristics, and pause length.…”

Section: Methodsmentioning

confidence: 99%

Towards an intelligent framework for multimodal affective data analysis

et al. 2015

View full text Add to dashboard Cite

An increasingly large amount of multimodal content is posted on social media websites such as YouTube and Facebook everyday. In order to cope with the growth of such so much multimodal data, there is an urgent need to develop an intelligent multi-modal analysis framework that can effectively extract information from multiple modalities. In this paper, we propose a novel multimodal information extraction agent, which infers and aggregates the semantic and affective information associated with user-generated multimodal data in contexts such as e-learning, e-health, automatic video content tagging and human-computer interaction. In particular, the developed intelligent agent adopts an ensemble feature extraction approach by exploiting the joint use of tri-modal (text, audio and video) features to enhance the multimodal information extraction process. In preliminary experiments using the eNTERFACE dataset, our proposed multi-modal system is shown to achieve an accuracy of 87.95%, outperforming the best state-of-the-art system by more than 10%, or in relative terms, a 56% reduction in error rate

show abstract

Section: Methodsmentioning

confidence: 99%

Towards an intelligent framework for multimodal affective data analysis

et al. 2015

View full text Add to dashboard Cite

show abstract

“…In these datasets, interactions include two interlocutors, who are recorded carrying out both structured and unstructured conversations. ere are numerous examples of such datasets, with a wide range of applications, such as speech recognition [15], behavior analysis [50], segmentation, emotion recognition [12] and depression detection [16]. Arguably, one of the most popular datasets of one-to-one interactions is SEMAINE [30].…”

Section: Related Workmentioning

confidence: 99%

The NoXi database: multimodal recordings of mediated novice-expert interactions

Cafaro

Wagner

Baur

et al. 2017

Proceedings of the 19th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

We present a novel multi-lingual database of natural dyadic noviceexpert interactions, named NoXi, featuring screen-mediated dyadic human interactions in the context of information exchange and retrieval. NoXi is designed to provide spontaneous interactions with emphasis on adaptive behaviors and unexpected situations (e.g. conversational interruptions). A rich set of audio-visual data, as well as continuous and discrete annotations are publicly available through a web interface. Descriptors include low level social signals (e.g. gestures, smiles), functional descriptors (e.g. turn-taking, dialogue acts) and interaction descriptors (e.g. engagement, interest, and uidity). CCS CONCEPTS•Information systems → Database design and models; Semistructured data; Data streams; •Human-centered computing → Systems and tools for interaction design; KEYWORDS A ective computing, multimodal corpora, multimedia databases ACM Reference format:

show abstract

“…The authors relied on the body movement to recognize the emotion instead of focusing on the overall body posture as a whole. Additionally, Caridakis et al proposed a method which utilized a Bayesian classifier to recognize emotion based on body gestures and speech [11]. In this literature, the authors collected multimodal data and then trained the data with a Bayesian classifier to recognize the emotion.…”

Section: Related Workmentioning

confidence: 99%

CEPP: Perceiving the Emotional State of the User Based on Body Posture

et al. 2017

View full text Add to dashboard Cite

Much research has been conducted in the area of face and gesture recognition in order to classify one's emotional state. Surprisingly, utilizing computerized algorithms which recognize emotional conditions based on body postures has not yet been systematically developed. In this paper, we propose a novel method, Computerized Emotion Perception based on Posture (CEPP), to determine the emotional state of the user. This method extracts features from body postures and estimates the emotional state by computing a similarity distance. With the proposed algorithm, we will provide new insights into automatically recognizing one's emotional state.

show abstract

Multimodal emotion recognition from expressive faces, body gestures and speech

Cited by 112 publications

References 23 publications

Towards an intelligent framework for multimodal affective data analysis

Towards an intelligent framework for multimodal affective data analysis

The NoXi database: multimodal recordings of mediated novice-expert interactions

CEPP: Perceiving the Emotional State of the User Based on Body Posture

Contact Info

Product

Resources

About