Predicting the Valence of a Scene from Observers’ Eye Movements

Abstract

Multimedia analysis benefits from understanding the emotional content of a scene in a variety of tasks such as video genre classification and content-based image retrieval. Recently, there has been an increasing interest in applying human bio-signals, particularly eye movements, to recognize the emotional gist of a scene such as its valence. In order to determine the emotional category of images using eye movements, the existing methods often learn a classifier using several features that are extracted from eye movements. Although it has been shown that eye movement is potentially useful for recognition of scene valence, the contribution of each feature is not well-studied. To address the issue, we study the contribution of features extracted from eye movements in the classification of images into pleasant, neutral, and unpleasant categories. We assess ten features and their fusion. The features are histogram of saccade orientation, histogram of saccade slope, histogram of saccade length, histogram of saccade duration, histogram of saccade velocity, histogram of fixation duration, fixation histogram, top-ten salient coordinates, and saliency map. We utilize machine learning approach to analyze the performance of features by learning a support vector machine and exploiting various feature fusion schemes. The experiments reveal that ‘saliency map’, ‘fixation histogram’, ‘histogram of fixation duration’, and ‘histogram of saccade slope’ are the most contributing features. The selected features signify the influence of fixation information and angular behavior of eye movements in the recognition of the valence of images.

Discovering gender differences in facial emotion recognition via implicit behavioral cues

“…While observed ER results are modest, they are still better or competitive with respect to prior eye and neural based ER approaches. Eye based features are found to achieve 52.5% valence recognition accuracy in [17], where emotions are induced in viewers by presenting a diverse types of emotional scenes to viewers as against emotional faces specifically employed in this work. Also, prior neural-based emotional studies [6], [24] achieve only around 60% valence recognition with lab-grade sensors.…”

“…Liu et al [5] perform ER with differential autoencoders, and attempt cross-modal ER based on shared EEG and eye-based representations. Tavakoli et al [17] perform valence (+ve vs -ve emotion) recognition using eye movements with an emphasis on evaluating eye-based features and their fusion, and achieve 52.5% accuracy with discriminative feature selection and a linear SVM.…”

“…Fig.4(b) suggests that when eye information is unavailable, females tend to concentrate on the mouth and nasal regions for emotional cues. Upon extracting the fixations and saccades, we adopted features used for recognizing valence (pleasant vs unpleasant emotions) in [17], namely, saccade orientation, top-ten salient coordinates, saliency map and histograms of a) saccade slope, b) saccade length, c) saccade velocity, 2. ERP data is plotted upside down as per convention.…”

Visual Attention Saccadic Models Learn to Emulate Gaze Patterns From Childhood to Adulthood

“…Wang et al [3] combined eye tracking with computational attention models in order to screen for mental diseases such as autism spectrum disorder (see also [4], [5]). Tavakoli et al [6] investigated the influence of eye-movement-based features to determine the valence of images.…”

Recognition of Advertisement Emotions With Application to Computational Advertising

“…In contrast, usercentric AR methods [14]- [16] estimate the stimulus-evoked emotion based on physiological changes observed in viewers (content consumers). Physiological signals indicative of emotions include pupillary dilation [26], eye-gaze patterns [9], [27] and neural activity [14], [15], [28]. Both content and user centric methods require labels denoting stimulus emotion, and such labels are compiled from annotators whose affective opinions are deemed acceptable [29], [30], given that emotion perception is highly subjective.…”