Building Emotional Machines: Recognizing Image Emotions Through Deep Neural Networks

Abstract

An image is a very effective tool for conveying emotions. Many researchers have investigated in computing the image emotions by using various features extracted from images. In this paper, we focus on two high level features, the object and the background, and assume that the semantic information of images is a good cue for predicting emotion. An object is one of the most important elements that define an image, and we find out through experiments that there is a high correlation between the object and the emotion in images. Even with the same object, there may be slight difference in emotion due to different backgrounds, and we use the semantic information of the background to improve the prediction performance. By combining the different levels of features, we build an emotion based feed forward deep neural network which produces the emotion values of a given image. The output emotion values in our framework are continuous values in the 2-dimensional space (Valence and Arousal), which are more effective than using a few number of emotion categories in describing emotions. Experiments confirm the effectiveness of our network in predicting the emotion of images.

Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias

“…While these approaches have obtained reasonable performance on such controlled emotion datasets, they have not yet considered predicting emotions from natural images as discussed in this paper. Most related to our work along the direction of recognizing emotions from natural images are the works of [63,38,30,43] which predict emotions from images crawled from Flickr and Instagram. As an example, the authors in [63] learn a CNN model to recognize emotions in natural images and performs reasonably well on the Deep Emotion dataset [63].…”

“…Recently, algorithms for object recognition and related tasks have become sufficiently proficient that new vision tasks beyond objects can now be pursued. One such task is to recognize emotions expressed by images which has gained momentum in last couple of years in both academia and industries [63,30,40,43,62,4]. Teaching machines to recognize diverse emotions is a very challenging problem with great application potential.…”

Assessing the Effectiveness of Automated Emotion Recognition in Adults and Children for Clinical Investigation

“…As we have seen from the above categorization of emotions, Ekman (2003); Ekman and Friesen (1978), a pioneer in the visual modality analysis of emotions, referred to facial expressions as primary cues for understanding emotions and sentiments. Facial expressions are a gateway into the human mind, emotion, and identity and, along with textual data, can provide important cues to better identify true affective states in the participants (Taggart et al, 2016;Kim et al, 2018). It can be crucial to understand facial characteristics when working with patients, especially patients who are unable to communicate in other ways, for example, when trying to assess emotions in children unable to selfreport information.…”

“…These algorithms detect facial landmarks and apply a set of rules based on psychological theories and statistical procedures to classify emotions (Li and Deng, 2018;Stöckli et al, 2018). Different algorithms, like AFFDEX and FACET, use distinct statistical procedures, facial databases, and facial landmarks to train the machine learning procedures and ultimately classify emotions (Kim et al, 2018). For all our experiments, we have used the AFFDEX algorithm.…”

Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis

“…tive subject of research to devise methods and tools to enable such applications [20,35,2]. Driven by the availability of large-scale annotated datasets [15,40] along with modern deep learning models, language sentiment analysis witnessed great improvements over the last few years [32].…”