Affect valence inference from facial action unit spectrograms

Abstract

The face provides an important channel for communicating affect valence, the positive or negative emotional charge of an experience. This paper addresses the challenging pattern recognition problem of assigning affect valence labels (positive, negative or neutral) to facial action sequences obtained from unsegmented videos coded using the Facial Action Coding System (FACS). The data were obtained from viewers watching eight short movies with each second of video labeled with self-reported valence and hand coded using FACS. We identify the most frequently occurring Facial Actions and propose the usefulness of a Facial Action Unit Spectrogram. We compare both generative and discriminative classifiers on accuracy and computational complexity: Support Vector Machines, Hidden Markov Models, Conditional Random Fields and Latent-Dynamic Conditional Random Fields. We conduct three tests of generalization with each model. The results provide a first benchmark for classification of self-report valences from spontaneous expressions from a large group of people (n=42). Success is demonstrated for increasing levels of generalization and discriminative classifiers are shown to significantly outperform generative classifiers over this large data set. We discuss the challenges encountered in dealing with a naturalistic dataset with sparse observations and its implications on the results.

Predicting online media effectiveness based on smile responses gathered over the Internet

“…Knowledge of likability and desire to view again are not only useful in advertisement copy-testing but could also be used to personalize the content viewers are shown when watching TV over the Internet using platforms such as Netflix or Hulu. In the case of humorous commercials, smile activity is a good measure of positive advertisement attitude or liking, and this can be measured continuously and unobtrusively from video images [10].…”

A feasibility study in using facial expressions analysis to evaluate player experiences

“…The use of these six basic expressions, as opposed to the more detailed Facial Action Coding System (FACS) [10] is a conscious decision due to the fact that FACS action unit recognition being still an open problem [15,14]. In the future, we will gradually investigate the feasibility of using FACS as FACS recognition improves.…”

Robust continuous prediction of human emotions using multiscale dynamic cues

“…Cruz et al [1] propose an approach that aligns the faces with Avatar Image Registration, and subsequently compute LPQ features. Mcdu↵ et al [9] predict valence using facial Action Unit spectrograms as features. In this study, we focus on mid-level dynamic features, extracted using di↵erent visual cues: head movements, face deformations and also global and local face appearance variations.…”