Fusing audio, visual and textual clues for sentiment analysis from multimodal content

Poria, Soujanya; Wang, Zhaoxia; Howard, Newton; Huang, Guang-Bin; Hussain, Amir

doi:10.1016/j.neucom.2015.01.095

Cited by 426 publications

(209 citation statements)

References 43 publications

Supporting

Mentioning

178

Contrasting

Unclassified

Order By: Relevance

“…A further subset of tweets (categorized as General) was collected from the stream, without any keyword filters, in order to further broaden the representative scope of our dataset. We additionally filtered out tweets containing external links or images, arguing that analysis of these multimodal tweets is a separate problem, belonging to the domain of Multimodal Sentiment Analysis (Poria et al, 2016;Soleymani et al, 2017). After the entire filtering process 4 was complete, we obtained 7,026 tweets across the different topics, which would be annotated with 5x coverage.…”

Section: Data Collectionmentioning

confidence: 99%

Sentiment Analysis: It’s Complicated!

Kenyon-Dean

Ahmed²,

Fujimoto

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

Sentiment analysis is used as a proxy to measure human emotion, where the objective is to categorize text according to some predefined notion of sentiment. Sentiment analysis datasets are typically constructed with gold-standard sentiment labels, assigned based on the results of manual annotations. When working with such annotations, it is common for dataset constructors to discard "noisy" or "controversial" data where there is significant disagreement on the proper label. In datasets constructed for the purpose of Twitter sentiment analysis (TSA), these controversial examples can compose over 30% of the originally annotated data. We argue that the removal of such data is a problematic trend because, when performing real-time sentiment classification of short-text, an automated system cannot know a priori which samples would fall into this category of disputed sentiment. We therefore propose the notion of a "complicated" class of sentiment to categorize such text, and argue that its inclusion in the short-text sentiment analysis framework will improve the quality of automated sentiment analysis systems as they are implemented in real-world settings. We motivate this argument by building and analyzing a new publicly available TSA dataset of over 7,000 tweets annotated with 5x coverage, named MTSA. Our analysis of classifier performance over our dataset offers insights into sentiment analysis dataset and model design, how current techniques would perform in the real world, and how researchers should handle difficult data.

show abstract

Section: Data Collectionmentioning

confidence: 99%

Sentiment Analysis: It’s Complicated!

Kenyon-Dean

Ahmed²,

Fujimoto

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

show abstract

“…Some recent work has also been conducted on fusing different modalities to detect emotions and polarity from videos [6], [7], [8]. This paper conducts extensive research on the different facets of this topic and aims to solve the following two questions: 1) Is a common framework useful for both multimodal emotion recognition and multimodal sentiment analysis?…”

Section: Introductionmentioning

confidence: 99%

Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

Poria

Chaturvedi

Wang

et al. 2016

2016 IEEE 16th International Conference on Data Mining (ICDM)

Self Cite

489

249

View full text Add to dashboard Cite

Abstract-Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

show abstract

“…While the visual and the audio modalities provide additional evidence that improves classification accuracy, we found the textual modality to have the greater impact on the result (Cambria and Hussain, 2015;Cambria et al, 2013c;Poria et al, 2015a;2015b).…”

Section: Introductionmentioning

confidence: 74%

Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis

Poria¹,

Wang²,

Gelbukh³

2015

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

Self Cite

454

199

View full text Add to dashboard Cite

We present a novel way of extracting features from short texts, based on the activation values of an inner layer of a deep convolutional neural network. We use the extracted features in multimodal sentiment analysis of short video clips representing one sentence each. We use the combined feature vectors of textual, visual, and audio modalities to train a classifier based on multiple kernel learning, which is known to be good at heterogeneous data. We obtain 14% performance improvement over the state of the art and present a parallelizable decision-level data fusion method, which is much faster, though slightly less accurate.

show abstract

Fusing audio, visual and textual clues for sentiment analysis from multimodal content

Cited by 426 publications

References 43 publications

Sentiment Analysis: It’s Complicated!

Sentiment Analysis: It’s Complicated!

Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis

Contact Info

Product

Resources

About