Audio–Visual Affective Expression Recognition Through Multistream Fused HMM

Zeng, Zhihong; Tu, Jilin; Pianfetti, B.; Huang, Thomas S.

doi:10.1109/tmm.2008.921737

Cited by 116 publications

(53 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The affect recognition studies [2,3] have found that many studies have employed supervised learning techniques for emotion recognition. Researchers have successfully used classification methods such as Bayesian classifiers [4], Hidden Markov Models (HMM) [15], and SVM [16,17] for affect recognition. Results in [6,10,11] and the classification techniques discussed in multimodal research surveys [32] showed that the performance improved in affect recognition by using SVM classifiers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Patwardhan¹

2017

MTI

View full text Add to dashboard Cite

This paper presents a multimodal emotion recognition method that uses a feature-level combination of three-dimensional (3D) geometric features (coordinates, distance and angle of joints), kinematic features such as velocity and displacement of joints, and features extracted from daily behavioral patterns such as frequency of head nod, hand wave, and body gestures that represent specific emotions. Head, face, hand, body, and speech data were captured from 15 participants using an infrared sensor (Microsoft Kinect). The 3D geometric and kinematic features were developed using raw feature data from the visual channel. Human emotional behavior-based features were developed using inter-annotator agreement and commonly observed expressions, movements and postures associated to specific emotions. The features from each modality and the behavioral pattern-based features (head shake, arm retraction, body forward movement depicting anger) were combined to train the multimodal classifier for the emotion recognition system. The classifier was trained using 10-fold cross validation and support vector machine (SVM) to predict six basic emotions. The results showed improvement in emotion recognition accuracy (The precision increased by 3.28% and the recall rate by 3.17%) when the 3D geometric, kinematic, and human behavioral pattern-based features were combined for multimodal emotion recognition using supervised classification.

show abstract

Section: Introductionmentioning

confidence: 99%

“…Researchers [11,15,36] have used facial and audio data fusion at the feature level for affect recognition. Hence, this paper uses concatenation of features from face, hand, head, body, audio, and the behavioral rule-based features to form a joint feature vector for multimodal emotion recognition system implementation.…”

Section: Introductionmentioning

confidence: 99%

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Patwardhan¹

2017

MTI

View full text Add to dashboard Cite

show abstract

“…Different cues of facial expression, speech, body gestures and context are used to recognize human affective states. More and more efforts are being made on multimodal fusion because integrating the information from multiple channels may lead to an improved performance of affect recognition [4]- [7].…”

Section: Introductionmentioning

confidence: 99%

“…And a Boltzmann chain can implement the same classification as the corresponding HMM with the same trellis topology. Unlike the MFHMM ( [4]) which fuses component HMMs together by connecting the hidden states of one HMM to the observation of another HMM, our method interconnects the hidden states of two Boltzmann chains with a correlation matrix. This topology is more natural for multimodal affect recognition.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Affect Recognition Using Boltzmann Zippers

Zhang

2013

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYThis letter presents a novel approach for automatic multimodal affect recognition. The audio and visual channels provide complementary information for human affective states recognition, and we utilize Boltzmann zippers as model-level fusion to learn intrinsic correlations between the different modalities. We extract effective audio and visual feature streams with different time scales and feed them to two component Boltzmann chains respectively. Hidden units of the two chains are interconnected to form a Boltzmann zipper which can effectively avoid local energy minima during training. Second-order methods are applied to Boltzmann zippers to speed up learning and pruning process. Experimental results on audio-visual emotion data recorded by ourselves in Wizard of Oz scenarios and collected from the SEMAINE naturalistic database both demonstrate our approach is robust and outperforms the state-of-the-art methods.

show abstract

“…In the end, the ranked classification results produced from each cluster are fused at the decision level. Other works in the literature have also applied hybrid fusion approaches that are different from ours [128,129,192,222,224]. Islam et al [223] propose a three-phase fusion process toward audio and visual modalities: fusion within a single modality, fusion across modalities in the feature level, and fusion on the decision level according to the reliability of each modality.…”

Section: At Which Level Should the Fusion Be Performed?mentioning

confidence: 99%

Integrating Deep Learning with Correlation-based Multimedia Semantic Concept Detection

Ha¹

View full text Add to dashboard Cite

Audio–Visual Affective Expression Recognition Through Multistream Fused HMM

Cited by 116 publications

References 29 publications

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Multimodal Affect Recognition Using Boltzmann Zippers

Integrating Deep Learning with Correlation-based Multimedia Semantic Concept Detection

Contact Info

Product

Resources

About