Deep Mul Timodal Learning for Emotion Recognition in Spoken Language

Abstract

In this paper, we present a novel deep multimodal framework to predict human emotions based on sentence-level spoken language. Our architecture has two distinctive characteristics. First, it extracts the high-level features from both text and audio via a hybrid deep multimodal structure, which considers the spatial information from text, temporal information from audio, and high-level associations from low-level handcrafted features. Second, we fuse all features by using a three-layer deep neural network to learn the correlations across modalities and train the feature extraction and fusion modules together, allowing optimal global fine-tuning of the entire structure. We evaluated the proposed framework on the IEMOCAP dataset. Our result shows promising performance, achieving 60.4% in weighted accuracy for five emotion categories.

Audio‐Textual Emotion Recognition Based on Improved Neural Networks

“…Compared with [20,31] based on the IMOCAP database, the model presented in this paper performs better, as shown in Table 2. For the modal fusing, the feature-level and decision-level fusion method are all useful.…”

“…In order to test the performance of multimodal emotion recognition model proposed in this paper, we compared it with other different models on the IEMOCAP database. Soujanya Gu et al [20] applied CNN-LSTM to process the speech date and CNNs for the textual features learning; finally, they integrated all features and trained them with a three-layer deep neural network. ey adopted the feature fusion method which we also referenced.…”

“…For the research on multimodal emotional analysis, [17][18][19][20] all applied CNNs were as a trainable feature extractor to extract textual, visual, or audio features. Poria et al [17] fed all the emotional features into the MKL (multiple kernel learning) classifier.…”

“…In the model of [19], the final CNN layer computed the weighted sum of all the information extracted from the attention input. After extracting features from CNN networks, Gu et al [20] made use of a three-layer deep neural network to fuse the multimodal features.…”

Sequential Embedding Induced Text Clustering, a Non-parametric Bayesian Approach

“…The encoder-decoder model was recently introduced in natural language processing and computer vision to model sequential data such as phrases [10,11,29,30] and videos [13]. It has shown great performance on a number of tasks including machine translation [6], question answering [25] and video description [13].…”