M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues

Abstract

We present M3ER, a learning-based method for emotion recognition from multiple input modalities. Our approach combines cues from multiple co-occurring modalities (such as face, text, and speech) and also is more robust than other methods to sensor noise in any of the individual modalities. M3ER models a novel, data-driven multiplicative fusion method to combine the modalities, which learn to emphasize the more reliable cues and suppress others on a per-sample basis. By introducing a check step which uses Canonical Correlational Analysis to differentiate between ineffective and effective modalities, M3ER is robust to sensor noise. M3ER also generates proxy features in place of the ineffectual modalities. We demonstrate the efficiency of our network through experimentation on two benchmark datasets, IEMOCAP and CMU-MOSEI. We report a mean accuracy of 82.7% on IEMOCAP and 89.0% on CMU-MOSEI, which, collectively, is an improvement of about 5% over prior work.

Emotion Recognition by Fusing Time Synchronous and Time Asynchronous Representations

“…"A", "T", and "V" refer to the audio, text, and video modalities respectively. Results from [6] are not included since the test setting was not clear. Results from [13] and [32] were not obtained using leave-one-speaker-out 10-fold CV and thus not directly comparable.…”

“…Although significant progress has been made [1][2][3], AER is still a challenging research problem since human emotions are inherently complex, ambiguous, and highly personal. Humans often express their emotions using multiple simultaneous approaches, such as voice characteristics, linguistic content, facial expressions, and body actions, which makes AER by nature a complex multimodal task [4][5][6]. Furthermore, due to the difficulties in data collection, publicly available datasets often do not have enough speakers to properly cover personal variations in emotion expression.…”

“…For instance, various types of acoustic features can be fused with text features derived either from pre-trained word embeddings [10,11] or from a jointly trained neural network component [12,13]. Context-dependent hierarchical fusion [14,15], multi-head attention mechanisms [13], and multiplicative fusion [6] have been applied to emotion recognition.…”

Deep Emotion Recognition in Dynamic Data using Facial, Speech and Textual Cues: A Survey

“…The studies focusing on multimodal fusion experiment on multimodal data for the purpose of accurate improvement of emotion recognition, but are short of the empirical evidence to prove the effectiveness of models when some of modalities are unavailable. Mittal et al [95] propose M3ER that utilizes Modality Check Step to replace unavailable modality with proxy feature and fuses multimodal features by multiplicative fusion module. M3ER is a promising technique but similarly lack of experiments in unimodality and bimodality.…”

Robust Latent Representations Via Cross-Modal Translation and Alignment

“…However, in general, most multi-modal fusion techniques require for the testing phase the simultaneous presence of all the modalities that were used during the model training phase [1]. This requirement becomes a severe limitation in case one or more sensors are missing or their signals are severely corrupted by noise during testing, unless such situations are explicitly handled by the modelling framework [8]. Thus, it would be desirable to improve the testing performance of individual modalities using other modalities during training [3][9] [10].…”