Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Abstract

In this paper, we extensively present our solutions for the MuSe-Stress sub-challenge and the MuSe-Physio sub-challenge of Multimodal Sentiment Challenge (MuSe) 2021. The goal of MuSe-Stress sub-challenge is to predict the level of emotional arousal and valence in a time-continuous manner from audio-visual recordings and the goal of MuSe-Physio sub-challenge is to predict the level of psycho-physiological arousal from a) human annotations fused with b) galvanic skin response (also known as Electrodermal Activity (EDA)) signals from the stressed people. The Ulm-TSST dataset which is a novel subset of the audio-visual textual Ulm-Trier Social Stress dataset that features German speakers in a Trier Social Stress Test (TSST) induced stress situation is used in both subchallenges. For the MuSe-Stress sub-challenge, we highlight our solutions in three aspects: 1) the audio-visual features and the biosignal features are used for emotional state recognition. 2) the Long Short-Term Memory (LSTM) with the self-attention mechanism is utilized to capture complex temporal dependencies within the feature sequences. 3) the late fusion strategy is adopted to further boost the model's recognition performance by exploiting complementary information scattered across multimodal sequences. Our proposed model achieves CCC of 0.6159 and 0.4609 for valence and arousal respectively on the test set, which both rank in the top 3. For the MuSe-Physio sub-challenge, we first extract the audiovisual features and the bio-signal features from multiple modalities. Then, the LSTM module with the self-attention mechanism, and the Gated Convolutional Neural Networks (GCNN) as well as the * Both authors contributed equally to this research.

Hybrid Multimodal Fusion for Humor Detection

“…However, these two types of fusion methods cannot effectively explore the inter-modality dynamics. Recently proposed fusion methods can be categorized into several types as follows, i.e., multi-view learning methods [32,41], word-level fusion methods [15,35,42], tensor fusion [24,40], and hybrid fusion [9,25]. These fusion techniques are effective in learning inter-modality dynamics compared to feature-level and decision-level fusion and show marked performance gains.…”

Emotion recognition with multi-modal peripheral physiological signals

“…Emotions play a central role in the human experience, exerting a profound influence on both physiological and psychological states. This influence holds promise for diverse applications, including Internet of Things (IoT) devices (Abdallah et al, 2020;Fodor et al, 2023), safe driving practices (Ma et al, 2021), software engineering (Fritz et al, 2014), and beyond. Research in this domain has sought to capture and interpret emotional states through a variety of signals.…”

DCNet: Weakly Supervised Saliency Guided Dual Coding Network for Visual Sentiment Recognition

“…Visual information in social media offers useful information and enables people to share their instant psychological and physiological status [11]. Visual sentiment recognition plays a considerable role in understanding the sentimental response when humans see specific visual content.…”