A robust multimodal approach for emotion recognition

Abstract-A complete emotional expression in natural faceto-face conversation typically contains a complex temporal course. In this paper, we propose a temporal course modelingbased error weighted cross-correlation model (TCM-EWCCM) for speech emotion recognition. In TCM-EWCCM, a TCMbased cross-correlation model (CCM) is first used to not only model the temporal evolution of the extracted acoustic and prosodic features individually but also construct the statistical dependencies among paired acoustic-prosodic features in different emotional states. Then, a Bayesian classifier weighting scheme named error weighted classifier combination is adopted to explore the contributions of the individual TCM-based CCM classifiers for different acoustic-prosodic feature pairs to enhance the speech emotion recognition accuracy. The results of experiments on the NCKU-CASC corpus demonstrate that modeling the complex temporal structure and considering the statistical dependencies as well as contributions among paired features in natural conversation speech can indeed improve the speech emotion recognition performance.

show abstract

“…Finally, combining (12), (7), and (5) into (4) yields (13) for emotional class w recognition using TCM-EWCCM:…”

Section: A Model Derivationmentioning

confidence: 99%

Emotion recognition of conversational affective speech using temporal course modeling-based error weighted cross-correlation model

Lin

Wei

et al. 2014

Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific

View full text Add to dashboard Cite

show abstract

“…Busso et al [2], presents a system for recognizing emotions through facial expressions displayed in live video streams and video sequences. However, works such as those presented in [3] suggest that developing a good methodology of emotional states characterization based on facial expressions, leads to more robust recognition systems. Facial Action Coding System (FACS) proposed by Ekman et al [1], is a comprehensive and anatomically based system that is used to measure all visually discernible facial movements in terms of atomic facial actions called Action Units (AUs).…”

Section: Introductionmentioning

confidence: 99%

“…As AUs are independent of interpretation, they can be used for any high-level decision-making process, including the recognition of basic emotions according to Emotional FACS (EM-FACS), the recognition of various affective states according to the FACS Affect Interpretation Database (FACSAID) introduced by Ekman et al [4], [5], [6], [7]. From the detected features, it is possible to estimate the emotion present in a particular subject, based on an analysis of estimated facial expression shape in comparision to a set of facial expressions of each emotion [3], [8], [9].…”

Section: Introductionmentioning

confidence: 99%

Facial Landmarking Localization for Emotion Recognition Using Bayesian Shape Models

García¹,

Valencia²,

Orozco³

2013

Computer Science &Amp; Information Technology ( CS &Amp; IT )

View full text Add to dashboard Cite

show abstract

“…In [10] and [11], prosody features from pitch and RMS energy are extracted as audio features, and the local preserving projection (LPP) of facial texture images are adopted as visual features. To reasonably integrate the information from speech and facial expressions, [12] proposes a tripled hidden Markov model (triple-HMM) based emotion recognition system modeling the correlation of three component HMMs that are based individually on upper face, lower face, prosodic dynamic behaviors, and allows unlimited state asynchrony between these streams. Moreover, in [10 -12] it has been showed that the audio visual multi-modal emotion recognition performs much better than the mono-modal emotion recognition, i.e.…”

Section: Introductionmentioning

confidence: 99%

Audio Visual Emotion Recognition Based on Triple-Stream Dynamic Bayesian Network Models

Jiang

Cui

Zhang

et al. 2011

Affective Computing and Intelligent Interaction

View full text Add to dashboard Cite

Abstract. We present a triple stream DBN model (T_AsyDBN) for audio visual emotion recognition, in which the two audio feature streams are synchronous, while they are asynchronous with the visual feature stream within controllable constraints. MFCC features and the principle component analysis (PCA) coefficients of local prosodic features are used for the audio streams. For the visual stream, 2D facial features as well 3D facial animation unit features are defined and concatenated, and the feature dimensions are reduced by PCA. Emotion recognition experiments on the eNERFACE'05 database show that by adjusting the asynchrony constraint, the proposed T_AsyDBN model obtains 18.73% higher correction rate than the traditional multi-stream state synchronous HMM (MSHMM), and 10.21% higher than the two stream asynchronous DBN model (Asy_DBN).

show abstract

A robust multimodal approach for emotion recognition

Cited by 46 publications

References 14 publications

Emotion recognition of conversational affective speech using temporal course modeling-based error weighted cross-correlation model

Emotion recognition of conversational affective speech using temporal course modeling-based error weighted cross-correlation model

Facial Landmarking Localization for Emotion Recognition Using Bayesian Shape Models

Audio Visual Emotion Recognition Based on Triple-Stream Dynamic Bayesian Network Models

Contact Info

Product

Resources

About