DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks

Abstract

This paper introduces a visual sentiment concept classification method based on deep convolutional neural networks (CNNs). The visual sentiment concepts are adjective noun pairs (ANPs) automatically discovered from the tags of web photos, and can be utilized as effective statistical cues for detecting emotions depicted in the images. Nearly one million Flickr images tagged with these ANPs are downloaded to train the classifiers of the concepts. We adopt the popular model of deep convolutional neural networks which recently shows great performance improvement on classifying largescale web-based image dataset such as ImageNet. Our deep CNNs model is trained based on Caffe, a newly developed deep learning framework. To deal with the biased training data which only contains images with strong sentiment and to prevent overfitting, we initialize the model with the model weights trained from ImageNet. Performance evaluation shows the newly trained deep CNNs model SentiBank 2.0 (or called DeepSentiBank) is significantly improved in both annotation accuracy and retrieval performance, compared to its predecessors which mainly use binary SVM classification models.

Affective Image Content Analysis: Two Decades Review and New Perspectives

“…As the extensions of SentiBank [10], DeepSentiBank [103] and MVSO [78] train the detectors for 2,089 and 4,342 English ANPs, respectively, using existing deep architecture like CaffeNet, and then, the sentiment polarity can be inferred. Using text parsing technology and lexicon-based sentiment analysis tools, the adjectives can be mapped into "positive" or "negative"; likewise, the polarity of an image is derived.…”

“…Based on a deep CNN model, the classifiers of the 1,200 ANP concepts are trained using Caffe. The newly trained deep model named DeepSentiBank [103] performs better than non-deep Sen-tiBank on sentiment prediction. Benefiting from transfer learning, Xu et al [30] transferred the parameters of a CNN trained on the large-scale dataset (ImageNet) to the task of predicting sentiments.…”

Sentiment Analysis from Images of Natural Disasters

“…More recently, the concept of emotion/sentiment analysis has been extended to more complex images involving multiple objects and background details [12,6,22,7]. For instance, Wang et al [23] rely on mid and low-level visual features along with textual information for sentiment analysis in social media images.…”

“…For instance, Wang et al [23] rely on mid and low-level visual features along with textual information for sentiment analysis in social media images. Chen et al [6] proposed DeepSentiBank, a deep convolutional neural network-based framework for sentiment analysis of social media images. To train the proposed deep model, around one million images with strong emotions have been collected from Flickr.…”

“…More recently, the concept of emotion recognition has been extended to relatively more complex images having multiple objects and background details. Thanks to the recent advances in deep learning, encouraging results have been recently obtained [6,18].…”