Joint Visual-Textual Sentiment Analysis with Deep Neural Networks

You, Quanzeng; Luo, Jiebo; Jin, Hailin; Yang, Jianchao

doi:10.1145/2733373.2806284

Cited by 73 publications

(33 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Studies have been extensively conducted on image analysis, as in SentiBank [3] and in its improved version Deep-SentiBank [4] that makes use of deep neural networks. They have been recently extended to the multi-lingual context [16] and to image+text sentiment analysis, as in [1,34]. However, studies on video are still lacking.…”

Section: Related Workmentioning

confidence: 99%

Deep Sentiment Features of Context and Faces for Affective Video Analysis

Baecchi

Uricchio

Bertini

et al. 2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Given the huge quantity of hours of video available on video sharing platforms such as YouTube, Vimeo, etc. development of automatic tools that help users nd videos that t their interests has attracted the attention of both scienti c and industrial communities. So far the majority of the works have addressed semantic analysis, to identify objects, scenes and events depicted in videos, but more recently a ective analysis of videos has started to gain more attention. In this work we investigate the use of sentiment driven features to classify the induced sentiment of a video, i.e. the sentiment reaction of the user. Instead of using standard computer vision features such as CNN features or SIFT features trained to recognize objects and scenes, we exploit sentiment related features such as the ones provided by , and features extracted from models that exploit deep networks trained on face expressions. We experiment on two recently introduced datasets: LIRIS-ACCEDE [2] and MEDIAEVAL-2015, that provide sentiment annotations of a large set of short videos. We show that our approach not only outperforms the current state-of-the-art in terms of valence and arousal classi cation accuracy, but it also uses a smaller number of features, requiring thus less video processing.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Sentiment Features of Context and Faces for Affective Video Analysis

Baecchi

Uricchio

Bertini

et al. 2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…Borth et al However, research on combining visual and textual elements in sentiment prediction from microblogs is far behind the single-element research. You et al [22] fine-tuned a CNN on a collection of images from Getty Image for visual sentiment analysis and trained a paragraph vector model for textual sentiment analysis. Cao et al [24,25], and Wang et al [23] established a dataset consisting of textual messages with related images extracted from Sina Weibo and performed sentiment analysis by combing the prediction results of using n-gram textual features and mid-level visual features [32].…”

Section: Related Workmentioning

confidence: 99%

“…Some researchers have concentrated on sentiment analysis of microblogs, which typically include a short text and one or more related images in each post. You et al [22] and Cao et al [23][24][25] employed both text and images to predict sentiment. The Many researchers have contributed to sentiment analysis of textual content or visual content.…”

Section: Introductionmentioning

confidence: 99%

“…Some researchers have concentrated on Algorithms 2016, 9, 41 3 of 11 sentiment analysis of microblogs, which typically include a short text and one or more related images in each post. You et al [22] and Cao et al [23][24][25] employed both text and images to predict sentiment. The former focused mainly on detecting sentiment from English-language social media, while the latter established a dataset extracted from a Chinese social network (Sina Weibo) and provided a benchmark for combining visual and textual sentiment prediction from Chinese social media.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks

et al. 2016

View full text Add to dashboard Cite

Sentiment analysis of online social media has attracted significant interest recently. Many studies have been performed, but most existing methods focus on either only textual content or only visual content. In this paper, we utilize deep learning models in a convolutional neural network (CNN) to analyze the sentiment in Chinese microblogs from both textual and visual content. We first train a CNN on top of pre-trained word vectors for textual sentiment analysis and employ a deep convolutional neural network (DNN) with generalized dropout for visual sentiment analysis. We then evaluate our sentiment prediction framework on a dataset collected from a famous Chinese social media network (Sina Weibo) that includes text and related images and demonstrate state-of-the-art results on this Chinese sentiment analysis benchmark.

show abstract

“…Related to our happiness study, sentiment analysis on textual contents, visual contents and multimedia contents has been addressed in previous works [11], [12]. However, sentiment analysis focuses more on the latent sentiments conveyed by the carriers such as images rather then inferring the state of users.…”

Section: Related Workmentioning

confidence: 99%

The effect of pets on happiness: A data-driven approach via large-scale social media

Yuan

You

et al. 2016

2016 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Abstract-Psychologists have demonstrated that pets have a positive impact on owners' happiness. For example, lonely people are often advised to have a dog or cat to quell their social isolation. Conventional psychological research methods of analyzing this phenomenon are mostly based on surveys or self-reported questionnaires, which are time-consuming and lack of scalability. Utilizing social media as an alternative and complimentary resource could potentially address both issues and provide different perspectives on this psychological investigation. In this paper, we propose a novel and effective approach that exploits social media to study the effect of pets on owners' happiness. The proposed framework includes three major components: 1) collecting user-level data from Instagram consisting of about 300,000 images from 2905 users; 2) constructing a convolutional neural network (CNN) for pets classification, and combined with timeline information, further identifying pet owners and the control group; 3) measuring the confidence score of happiness by detecting and analyzing selfie images. Furthermore, various factors of demographics are employed to analyze the fine-grained effects of pets on happiness. Our experimental results demonstrate the effectiveness of the proposed approach and we believe that this approach can be applied to other related domains as a large-scale, high-confidence methodology of user activity analysis through social media.

show abstract

Joint Visual-Textual Sentiment Analysis with Deep Neural Networks

Cited by 73 publications

References 12 publications

Deep Sentiment Features of Context and Faces for Affective Video Analysis

Deep Sentiment Features of Context and Faces for Affective Video Analysis

Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks

The effect of pets on happiness: A data-driven approach via large-scale social media

Contact Info

Product

Resources

About