Latent trees for estimating intensity of Facial Action Units

Abstract

This paper is about estimating intensity levels of Facial Action Units (FAUs) in videos as an important step toward interpreting facial expressions. As input features, we use locations of facial landmark points detected in video frames. To address uncertainty of input, we formulate a generative latent tree (LT) model, its inference, and novel algorithms for efficient learning of both LT parameters and structure. Our structure learning iteratively builds LT by adding either a new edge or a new hidden node to LT, starting from initially independent nodes of observable features. A graph-edit operation that increases maximally the likelihood and minimally the model complexity is selected as optimal in each iteration. For FAU intensity estimation, we derive closed-form expressions of posterior marginals of all variables in LT, and specify an efficient bottom-up/topdown inference. Our evaluation on the benchmark DISFA and ShoulderPain datasets, in subject-independent setting, demonstrate that we outperform the state of the art, even under significant noise in facial landmarks. Effectiveness of our structure learning is demonstrated by probabilistically sampling meaningful facial expressions from the LT.

A Framework for Joint Estimation and Guided Annotation of Facial Action Unit Intensity

“…Note, however, that most of the existing methods perform the intensity estimation independently for each AU. Only recently, several methods for joint estimation of the AU intensity have been proposed [24,15,13]. The main motivation for this approach is that by joint modeling of AUs, the resulting AU intensity classifiers are more robust to the (highly) imbalanced intensity levels (within and between AUs), and non-additive AU combinations.…”

Joint Facial Action Unit Detection and Feature Fusion: A Multi-Conditional Learning Approach

“…The dependencies among observed features and multiple AUs are modeled via latent variables. Nevertheless, [12], [27] lack the classification power of the discriminative models. On the other hand, the models in [16], [15], [17], [13] are defined in a fully discriminative framework.…”

“…Due to the Markov assumptions while learning the network of the co-occurred AUs, this model can handle only local dependencies between pairs of AUs. The authors in [27] propose a generative latent tree algorithm for AU intensity estimation. The dependencies among observed features and multiple AUs are modeled via latent variables.…”

“…To the best of our knowledge, there are only few works that perform joint detection of AUs [12], [27], [16], [15], [17], [13]. [12] proposed a generative framework based on DBNs to model the semantics of different AUs.…”

Projective complex matrix factorization for facial expression recognition

“…Intensive techniques have been proposed for FER problem [6][7][8][9][10][11][12][13][14]. Much attention is put on the facial action coding system (FACS) approach which attempts to decompose facial expressions into varied action units and facial expression could be recognized based on the mixture of action units [6,7].…”