EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings

Abstract

Despite remarkable advances in emotion recognition, they are severely restrained from either the essentially limited property of the employed single modality, or the synchronous presence of all involved multiple modalities. Motivated by this, we propose a novel crossmodal emotion embedding framework called EmoBed, which aims to leverage the knowledge from other auxiliary modalities to improve the performance of an emotion recognition system at hand. The framework generally includes two main learning components, i. e., joint multimodal training and crossmodal training. Both of them tend to explore the underlying semantic emotion information but with a shared recognition network or with a shared emotion embedding space, respectively. In doing this, the enhanced system trained with this approach can efficiently make use of the complementary information from other modalities. Nevertheless, the presence of these auxiliary modalities is not demanded during inference. To empirically investigate the effectiveness and robustness of the proposed framework, we perform extensive experiments on the two benchmark databases RECOLA and OMG-Emotion for the tasks of dimensional emotion regression and categorical emotion classification, respectively. The obtained results show that the proposed framework significantly outperforms related baselines in monomodal inference, and are also competitive or superior to the recently reported systems, which emphasises the importance of the proposed crossmodal learning for emotion recognition.

Robust Latent Representations Via Cross-Modal Translation and Alignment

“…It is to be noted that the dataset provides separate features for arousal and valence. As in [9][16], to compensate for the delay in annotation, we shift the ground-truth labels back in time by 2.4 s. This dataset is ideal for our objective, since the uni-modal performance of audio and video features varies considerably for arousal and valence, as reported in [9] and confirmed by our experiments (see Table 1). As in the AVEC 2016 challenge, we use the Concordance Correlation Coefficient (CCC) (eq.…”

“…In order to identify the stronger and weaker modalities, we first assess the unimodal performances of audio, video-geometric and video-appearance features for arousal and valence using a regressor similar to [9]. The regressor consists of 4 single time-step GRU-RNN layers, each made up of 120 neurons, followed by a linear layer and trained using the MSE loss.…”

“…However, the modalities considered in these methods are images of equal size and the uni-modal networks have the same architecture, thus preventing their direct application to distinct modalities like audio, video and text, whose feature types and dimensionality differ. A few works have been proposed to address this problem [9][11] [12]. For sentiment analysis, a sequence-to-sequence network with cyclic translation across modalities generates an intermediate representation that is robust to missing modalities during testing [11].…”

“…In contrast, we aim to explicitly improve the weaker modality using the stronger modality during training. A joint audiovisual training and cross-modal triplet loss can be used to develop a face/speech emotion recognition system using multi-modal training [9]. However, in such a system the weaker modality may degrade the performance of the stronger modality [9].…”

“…In this section, we compare the performance of SEW with other unimodal methods [16][17][18] [19] and a state-of-the-art cross-modal knowledge transfer method [9]. We describe the dataset, the evaluation metrics, the details about the architecture and training, and present an ablation study, which quantifies the contributions of different parts of SEW.…”