An End-To-End Emotion Recognition Framework Based on Temporal Aggregation of Multimodal Information

Abstract

Humans express and perceive emotions in a multimodal manner. The multimodal information is intrinsically fused by the human sensory system in a complex manner. Emulating a temporal desynchronisation between modalities, in this paper, we design an end-to-end neural network architecture, called TA-AVN, that aggregates temporal audio and video information in an asynchronous setting in order to determine the emotional state of a subject. The feature descriptors for audio and video representations are extracted using simple Convolutional Neural Networks (CNNs), leading to real-time processing. Undoubtedly, collecting annotated training data remains an important challenge when training emotion recognition systems, both in terms of effort and expertise required. The proposed approach solves this problem by providing a natural augmentation technique that allows achieving a high accuracy rate even when the amount of annotated training data is limited. The framework is tested on three challenging multimodal reference datasets for the emotion recognition task, namely the benchmark datasets CREMA-D and RAVDESS, and one dataset from the FG2020's challenge related to emotion recognition. The results prove the effectiveness of our approach and our end-to-end framework achieves state-of-the-art results on the CREMA-D and RAVDESS datasets.INDEX TERMS Emotion recognition, multimodal data, audiovisual information, augmentation techniques, convolutional neural network, real-time processing.

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

“…That is different from late fusion of modalities [27], [28] or building temporal features to extract global information by assuming that emotions are expressed simultaneously [26]. Late fusion is favorably applied by concatenating the learned features of all modalities in [27], [28] or with a pairwise scheme in [26]. Instead, the authors of M3ER [29] propose a data-driven multiplicative fusion method to combine the modalities, which learns to emphasize the more reliable cues and suppresses the others by integrating Canonical Correlation Analysis as a pre-processing step.…”

“…Early works adapt classifiers like SVMs, Linear and Logistic Regression [39], [40] while, by the time bigger datasets were developed, deep learning architectures were also explored. For example, [27] is based on CNNs, and [26], [28] use RNNs. Some recent studies [41], [14], [16] adopt Transformers.…”

“…Ghaleb et al [42] apply deep metric learning in which a LSTM component models the variations of the emotions as a function of time. That is different from late fusion of modalities [27], [28] or building temporal features to extract global information by assuming that emotions are expressed simultaneously [26]. Late fusion is favorably applied by concatenating the learned features of all modalities in [27], [28] or with a pairwise scheme in [26].…”

“…It is applied here only at inference via the concatenation of learned feature representations. This is different from the MER state-of-the-art (SOTA) applying data fusion both in training and testing [27], [26], [28], [29]. v) Data labelling.…”

Mining Inconsistent Emotion Recognition Results With the Multidimensional Model

“…Classifiers are usually built on one of the known artificial intelligence tools and algorithms, including decision trees, neural networks, Bayesian networks, linear discriminate analysis, linear logistic regression, Support Vector Machine, Hidden Markov Models [28], and lately also convolutional networks [26]. Deep learning becomes a new trend in emotion recognition, especially applied to facial expression recognition [30,14,29]. Depending on the classification method, input channels and selected features, the accuracy of affect recognition differs significantly, sometimes achieving more than 90 percent, but mostly in a cross-validation scheme on a single dataset.…”