Attention Driven Fusion for Multi-Modal Emotion Recognition

Abstract

Deep learning has emerged as a powerful alternative to hand-crafted methods for emotion recognition on combined acoustic and text modalities. Baseline systems model emotion information in text and acoustic modes independently using Deep Convolutional Neural Networks (DCNN) and Recurrent Neural Networks (RNN), followed by applying attention, fusion, and classification. In this paper, we present a deep learning-based approach to exploit and fuse text and acoustic data for emotion classification. We utilize a SincNet layer, based on parameterized sinc functions with band-pass filters, to extract acoustic features from raw audio followed by a DCNN. This approach learns filter banks tuned for emotion recognition and provides more effective features compared to directly applying convolutions over the raw speech signal. For text processing, we use two branches (a DCNN and a Bidirection RNN followed by a DCNN) in parallel where cross attention is introduced to infer the N-gram level correlations on hidden representations received from the Bi-RNN. Following existing state-of-the-art, we evaluate the performance of the proposed system on the IEMOCAP dataset. Experimental results indicate that the proposed system outperforms existing methods, achieving 3.5% 1 improvement in weighted accuracy.

Pause-Encoded Language Models for Recognition of Alzheimer’s Disease and Emotion

“…To our knowledge, the best reported accuracy of using textual features only on this dataset was 70.8% [33]. Finetuning the off-the-shelf RoBERTa on word transcripts only (without pauses) achieved a better performance.…”

“…We used IEMOCAP dataset [29], a benchmark dataset containing 12 hours of speech from 10 professional actors. Following the literature [30,31,32,33], we extracted 5531 utterances of four emotion types from the dataset: 1636 happy (also including excited), 1084 sad, 1103 angry, and 1708 neutral. The utterances were forced aligned using the P2FA forced aligner.…”

Multimodal Emotion Recognition using Deep Learning

“…[ Priyasad et al] [64] presented a deep learning-based approach to protect codes that are characteristic emotion. Through a SincNet layer, band-pass filtering technique and neural net, the researchers managed to extract acoustic Identify applicable funding agency here.…”

Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation

“…There is a recent interest on attention-based SER models for higher accuracy [8,9,12]. However, those attention mechanisms can only be calculated with a preset granularity which may not adapt dynamically to different areas of interest in spectrogram.…”

“…For example, a psychologist can design a treatment plan according to the emotions hidden/expressed in the patient's speech. Deep learning has accelerated the progress of recognizing human emotions from speech [4][5][6][7][8][9], but there are still deficiencies in the research of SER, such as data shortage and insufficient model accuracy.…”