Emotion Recognition in the Wild

Ringeval, Fabien; Amiriparian, Shahin; Eyben, Florian; Scherer, Klaus R.; Schuller, Björn

doi:10.1145/2663204.2666271

Cited by 21 publications

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methods can be broadly categorized as either feature level fusion or decision level fusion (Wu et al, 2014). The former is generally carried out by concatenating feature vectors from different modalities (Metallinou et al, 2012;Kim and Clements, 2015), while the latter involves developing indepedent unimodal predictive models and then aggregating the predictions from each modality (Ringeval et al, 2014;Sahoo and Routray, 2016). A combination of both is also possible, for instance Metallinou et al (2012) first adopts feature level fusion with different weights assigned to audio and video modalities followed by model level fusion to learn a joint representation from multiple modalities.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Affect Models: An Investigation of Relative Salience of Audio and Visual Cues for Emotion Prediction

Wu¹,

Dang²,

Sethu³

et al. 2021

Front. Comput. Sci.

View full text Add to dashboard Cite

People perceive emotions via multiple cues, predominantly speech and visual cues, and a number of emotion recognition systems utilize both audio and visual cues. Moreover, the perception of static aspects of emotion (speaker's arousal level is high/low) and the dynamic aspects of emotion (speaker is becoming more aroused) might be perceived via different expressive cues and these two aspects are integrated to provide a unified sense of emotion state. However, existing multimodal systems only focus on single aspect of emotion perception and the contributions of different modalities toward modeling static and dynamic emotion aspects are not well explored. In this paper, we investigate the relative salience of audio and video modalities to emotion state prediction and emotion change prediction using a Multimodal Markovian affect model. Experiments conducted in the RECOLA database showed that audio modality is better at modeling the emotion state of arousal and video for emotion state of valence, whereas audio shows superior advantages over video in modeling emotion changes for both arousal and valence.

show abstract