Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing

Abstract

As an important research issue in affective computing community, multi-modal emotion recognition has become a hot topic in the last few years. However, almost all existing studies perform multiple binary classification for each emotion with focus on complete time series data. In this paper, we focus on multi-modal emotion recognition in a multi-label scenario. In this scenario, we consider not only the label-to-label dependency, but also the feature-to-label and modality-to-label dependencies. Particularly, we propose a heterogeneous hierarchical message passing network to effectively model above dependencies. Furthermore, we propose a new multi-modal multi-label emotion dataset based on partial time-series content to show predominant generalization of our model. Detailed evaluation demonstrates the effectiveness of our approach.

Multi-View Multi-Label Fine-Grained Emotion Decoding From Human Brain Activity

“…[19] employed adversarial learning and combined adversarial loss and multi-label supervised loss to achieve multi-label emotion tagging for video data. [20] exploited Graph Neural Networks (GNNs) with heterogeneous hierarchical message passing for multi-modal multi-label emotion classification with textual, visual and acoustic modalities. However, these studies all belong to coarse-grained emotion classification which involve only 12 emotion categories at most [20].…”

“…[20] exploited Graph Neural Networks (GNNs) with heterogeneous hierarchical message passing for multi-modal multi-label emotion classification with textual, visual and acoustic modalities. However, these studies all belong to coarse-grained emotion classification which involve only 12 emotion categories at most [20]. It is difficult for these studies to truly reflect the complex emotion categories of human.…”

MMPosE: Movie-Induced Multi-Label Positive Emotion Classification Through EEG Signals

“…Aiming at facial expression recognition, Li and Deng [33] proposed a new deep manifold learning network to learn discriminative features of multi-label expressions. Moreover, Zhang et al [34] focused on multi-modal emotion recognition in a multi-label scenario.…”

Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers

“…To improve the decoding efficiency of autoregressive transformers, bidirectional generative transformers have been proposed [5,13,56]. Contrary to autoregressive models that predict a single consecutive token at each step, a bidirectional transformer learns to predict multiple masked tokens at once based on the previously generated context.…”