Semantic Audiovisual Data Fusion for Automatic Emotion Recognition

Abstract

The paper describes a novel technique for the recognition of emotions from multimodal data. We focus on the recognition of the six prototypic emotions. The results from the facial expression recognition and from the emotion recognition from speech are combined using a bi-modal multimodal semantic data fusion model that determines the most probable emotion of the subject. Two types of models based on geometric face features for facial expression recognition are being used, depending on the presence or absence of speech. In our approach we define an algorithm that is robust to changes of face shape that occur during regular speech. The influence of phoneme generation on the face shape during speech is removed by using features that are only related to the eyes and the eyebrows. The paper includes results from testing the presented models.

Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

“…[15] developed a facial expression coding system (FACS) and six facial expressions that are able to provide sufficient clues to detect emotions. Recent studies on speech-based emotion analysis [4] have focused on identifying several acoustic features. One of the earliest works on fusing audio-visual emotion recognition [16] showed that a bimodal system yields higher accuracy than any unimodal system.…”

“…People, however, are gradually shifting from text to video to express their opinion about a product or service, as it is now much easier and faster for them to produce and share multimodal content [3]. For the same reasons, potential customers are now more inclined to browse for video reviews of the product they are interested in, rather than looking for lengthy written reviews [4]. Another reason for doing this is that, while trustable written reviews are quite hard to find, searching for good video reviews is as easy as typing the name of the product on YouTube and choosing the clips with more views [5].…”

Towards an intelligent framework for multimodal affective data analysis

“…Processing Classification algorithm Accuracy Lanitis et al [11] Appearance Model Distance-based 74% Cohen et al [16] Appearance Model Bayesian network 83% Mase [13] Optical flow kNN 86% Rosenblum et al [17] Recent studies on speech-based emotion analysis [12,[21][22][23][24][25][26][27] have focused on identifying several acoustic features such as fundamental frequency (pitch), intensity of utterance [15], bandwidth, and duration. The speaker-dependent approach gives much better results than the speaker-independent approach, as shown by the excellent results of Navas et al [29], where about 98% accuracy was achieved by using the Gaussian mixture model (GMM) as a classifier, with prosodic, voice quality as well as Mel frequency cepstral coefficient (MFCC) employed as speech features.…”

“…The Active Appearance Model [11,12] and Optical Flow-based techniques [13] are common approaches that use FACS to understand expressed facial expressions. Exploiting AUs as features, kNN, Bayesian networks, hidden Markov models (HMM) and artificial neural networks (ANN) [14] have been used by many researchers to infer emotions from facial expressions.…”

“…A number of studies favor decision-level fusion as the preferred method of data fusion because errors from different classifiers tend to be uncorrelated and the methodology is feature-independent [66]. Bimodal fusion methods have been proposed in numerous instances [12,67,68], but optimal information fusion configurations remain elusive.…”