Multimodal sentiment analysis using hierarchical fusion with context modeling

Majumder, Navonil; Hazarika, Devamanyu; Gelbukh, Alexander; Wang, Zhaoxia; Poria, Soujanya

doi:10.1016/j.knosys.2018.07.041

Cited by 289 publications

(131 citation statements)

References 45 publications

Supporting

Mentioning

128

Contrasting

Order By: Relevance

“…We compare HFFN with following multimodal algorithms: RMFN (Liang et al, 2018a), MFN (Zadeh et al, 2018a), MCTN (Pham et al, 2019), BC-LSTM (Poria et al, 2017b), TFN , MARN (Zadeh et al, 2018b), LMF ), MFM (Tsai et al, 2019, MR-RF (Barezi et al, 2018), FAF (Gu et al, 2018b), RAVEN (Wang et al, 2019), GMFN (Zadeh et al, 2018c), Memn2n (Sukhbaatar et al, 2015), MM-B2 , CHFusion (Majumder et al, 2018), SVM Trees (Rozgic et al, 2012), CMN , C-MKL (Poria et al, 2016b) and CAT-LSTM (Poria et al, 2017c).…”

Section: Comparison With Baselinesmentioning

confidence: 99%

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Mai¹,

Hu²,

Xing³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

We propose a general strategy named 'divide, conquer and combine' for multimodal fusion. Instead of directly fusing features at holistic level, we conduct fusion hierarchically so that both local and global interactions are considered for a comprehensive interpretation of multimodal embeddings. In the 'divide' and 'conquer' stages, we conduct local fusion by exploring the interaction of a portion of the aligned feature vectors across various modalities lying within a sliding window, which ensures that each part of multimodal embeddings are explored sufficiently. On its basis, global fusion is conducted in the 'combine' stage to explore the interconnection across local interactions, via an Attentive Bi-directional Skipconnected LSTM that directly connects distant local interactions and integrates two levels of attention mechanism. In this way, local interactions can exchange information sufficiently and thus obtain an overall view of multimodal information. Our method achieves state-ofthe-art performance on multimodal affective computing with higher efficiency.

show abstract

Section: Comparison With Baselinesmentioning

confidence: 99%

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Mai¹,

Hu²,

Xing³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Then these features were concatenated into a long vector fed into SVM. Majumder, Hazarika, Gelbukh, Cambria, and Poria (2018) proposed a novel feature-level fusion strategy which proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities.…”

Section: Textual Sentiment Recognitionmentioning

confidence: 99%

The sentiment recognition of online users based on DNNs multimodal fusion

Fan

Zhu

2019

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Online users are more willing to express their opinions and emotions on social platforms in online public opinion events, through combing texts and images rather than texts alone. Facing the change of emotional expressions, it is extremely vital to recognize the sentiments of online users in online public opinion events in an appropriate way. In this paper, we propose a novel Deep Neural Networks (DNNs) model to recognize online users' sentiments in online public opinion events by analyzing the sentiments of texts and attached images together. Also, we compare two fusion strategies, feature‐level fusion and decision‐level fusion, to combine the affective information from texts and images. In feature‐level fusion, fine‐tuned Convolutional Neural Networks (CNNs) and Bidirectional Long Short‐Term Memory (BiLSTMs) are employed to extract visual and textual features respectively. Then the features are concatenated and fed to a DNNs system to complete classification. In decision‐level fusion, a rule is employed to fuse the output from unimodality, generating final predicted labels. Experimental results showed that the proposed DNNs multimodal model achieved better performance than unimodal sentiment recognition model. In fusion strategy, the feature‐level fusion performed better in our experiments.

show abstract

“…Earlier marketing theories have believed in two types of strategies, "The Push" strategy that highlights pushing the produce/service towards the customers by means of extensive and aggressive marketing and promotion, in contrast to, "The Pull" strategy that Sentiment Analysis-A tool for Data Mining in Big Data Analytics Girisha Moorjani, Lipsa Sadath signifies the importance of attracting the customer towards what is being provided [19]. In today's dynamic business scenario, it becomes imperative for banks to utilize the pull strategy considering the number of substitutes available out there and for this purpose it is vital for them to know the true need of their customers.…”

Section: Evolution Of Sentiment Analysismentioning

confidence: 99%

Sentiment Analysis- A tool for Data Mining in Big Data Analytics

Moorjani¹,

Sadath²

2019

IJITEE

View full text Add to dashboard Cite

With the current business environment and rapid changes in technology, the amount of data produced is increasing as each day passes. This huge collection of data is what can make or break such institutions, so it is vital for such a sector to efficiently utilize the data generated. Effective tools and analyses are required to make sure that this data is comprehended and organized in such a manner that it can be used for the tasks at hand. The challenge faced here is knowing how to extract and use the data to the benefit of the business world. The objective of understanding the underlying emotion displayed in each opinion that is voiced out is a huge exercise. Through this paper an attempt has been made to understand how the gap between consumers and providers can be bridged by analyzing secondary data through Sentiment Analysis tool. This research proposes a framework CSA (Continuous Sentiment Analysis) to repeatedly analyze the sentiments from customers highlighting the purpose of one such attempt to capture the tone of the message. This method of “Sentiment Analysis”- a fairly new field uses Natural Language Processing (NLP) in order to give meaning to the abundant data available at hand.

show abstract

Multimodal sentiment analysis using hierarchical fusion with context modeling

Cited by 289 publications

References 45 publications

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

The sentiment recognition of online users based on DNNs multimodal fusion

Sentiment Analysis- A tool for Data Mining in Big Data Analytics

Contact Info

Product

Resources

About