Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis

Abstract

In recent times, multi-modal analysis has been an emerging and highly sought-after field at the intersection of natural language processing, computer vision, and speech processing. The prime objective of such studies is to leverage the diversified information, (e.g., textual, acoustic and visual), for learning a model. The effective interaction among these modalities often leads to a better system in terms of performance. In this paper, we introduce a recurrent neural network based approach for the multi-modal sentiment and emotion analysis. The proposed model learns the intermodal interaction among the participating modalities through an auto-encoder mechanism. We employ a context-aware attention module to exploit the correspondence among the neighboring utterances. We evaluate our proposed approach for five standard multi-modal affect analysis datasets. Experimental results suggest the efficacy of the proposed model for both sentiment and emotion analysis over various existing state-of-the-art systems.

Unsupervised Multimodal Language Representations using Convolutional Autoencoders

“…These embeddings have been used to train a Logistic Regression model for the two downstream tasks of Sentiment Analysis and Emotion Recognition. Following the existing literature [8,2,5,15,3,16,10,17,9,13], we report the binary accuracy and weighted averaged f1 metrics on sentiment for MOSEI, and on each of the 4 emotions of IEMOCAP in an one-vs-all manner.…”

“…In this paper we follow a widely used approach for basic unimodal feature extraction and multimodal alignment, similar to the one in various proposed methods, such as [8,2,5,15,3,16,10,17,9,13]. More specifically, after extracting the features on each modality(visual, textual and aural), the procedure of word-level alignment that was firstly used for this task in [17], is performed.…”

“…Different types of fusion approaches, such as early fusion [1], memory fusion [2,3], multistage fusion [4] and tensor fusion [5] have been examined. Modifications on LSTM architectures for multiview learning [6] or context-dependent analysis [7] have also been proposed, as well as the concept of attention on recurrent networks [8], context-aware attention [9] and some transformer architectures [10] have all been researched in depth.…”

Urdu Sentiment Analysis via Multimodal Data Mining Based on Deep Learning Algorithms

“…The experimental results of their research show that if the multimodal features are contextually integrated, then it can result in better performance (up to an accuracy level of 91.39%) as compared to the features that are extracted by a unimodal (having an accuracy value of up to 89.24%). Chauhan et al [7] have devised a multimodal emotions analysis model based on the recurrent neural network (RNN). This model also learns the interaction within different participating models by using an auto-encoder algorithm-based mechanism.…”

Improving Aspect-based Sentiment Analysis with Gated Graph Convolutional Networks and Syntax-based Regulation

“…Sentiment analysis has been studied under different settings in the literature (e.g., sentence-level, aspect-level, cross-domain) Chauhan et al, 2019;Hu et al, 2019). For ABSA, the early works have performed feature engineering to produce useful features for the statistical classification models (e.g., SVM) (Wagner et al, 2014).…”