Contextual Inter-modal Attention for Multi-modal Sentiment Analysis

Abstract

Multi-modal sentiment analysis offers various challenges, one being the effective combination of different input modalities, namely text, visual and acoustic. In this paper, we propose a recurrent neural network based multi-modal attention framework that leverages the contextual information for utterance-level sentiment prediction. The proposed approach applies attention on multi-modal multi-utterance representations and tries to learn the contributing features amongst them. We evaluate our proposed approach on two multi-modal sentiment analysis benchmark datasets, viz. CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) corpus and the recently released CMU Multi-modal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) corpus. Evaluation results show the effectiveness of our proposed approach with the accuracies of 82.31% and 79.80% for the MOSI and MO-SEI datasets, respectively. These are approximately 2 and 1 points performance improvement over the state-of-the-art models for the datasets.

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

“…Multimodal sentiment analysis provides an opportunity to learn interactions between different modalities. Similar to approaches mentioned for intermodal attention in Ghosal et al [10], we propose a method to learn cross-interaction vectors. For a pair of Text (H T ) and Video (H V ) modalities, co-attention matrix (M T V ∈R u×u ) can be defined as:…”

“…In our experiments, we used same features mentioned in Ghosal et al [10]. Specifically, for CMU-MOSEI dataset, we used Glove embeddings for word features, Facets 2 for visual features and CovaRep [16] for acoustic features.…”

“…[5] 76.5 73. 4 Zadeh et al [7] 76.9 77.0 Georgiou et al [9] 76.9 76.9 Poria et al [2] 77.64 -Ghosal et al [10] 82.31 80.69 Ghosal et al [10] 79.80 -Sun et al [4] 80 [7], ( ¦ ) results are obtained on CMU-MOSEI dataset after excluding the utterances with sentiment score of 0. We mention the results of proposed model with this setup in the parenthesis.…”

“…Methods that jointly learn the interactions between two or three modalities [3,4], and 3. Methods that explicitly learn contributions from these unimodal and cross modal cues, typically using attention based techniques [5,6,7,8,9,10].…”

“…Most of the existing approaches propose either fusion at different granularities [3,9] or use a cross interaction block that couple the features from different modalities [10,6].…”