Deep Structured Learning for Facial Action Unit Intensity Estimation

Abstract

We consider the task of automated estimation of facial expression intensity. This involves estimation of multiple output variables (facial action units -AUs) that are structurally dependent. Their structure arises from statistically induced co-occurrence patterns of AU intensity levels. Modeling this structure is critical for improving the estimation performance; however, this performance is bounded by the quality of the input features extracted from face images. The goal of this paper is to model these structures and estimate complex feature representations simultaneously by combining conditional random field (CRF) encoded AU dependencies with deep learning. To this end, we propose a novel Copula CNN deep learning approach for modeling multivariate ordinal variables. Our model accounts for ordinal structure in output variables and their non-linear dependencies via copula functions modeled as cliques of a CRF. These are jointly optimized with deep CNN feature encoding layers using a newly introduced balanced batch iterative training algorithm. We demonstrate the effectiveness of our approach on the task of AU intensity estimation on two benchmark datasets. We show that joint learning of the deep features and the target output structure results in significant performance gains compared to existing deep structured models for analysis of facial expressions.

Multi-Instance Dynamic Ordinal Random Fields for Weakly Supervised Facial Behavior Analysis

“…It is also worth mentioning recent works on Deep Learning for Action Unit detection [47] and Intensity Estimation [48], [11]. Although these models have a high modelling power, the reported results have not shown significant improvements compared to traditional shallow methods using hand-crafted features.…”

“…Although these models have a high modelling power, the reported results have not shown significant improvements compared to traditional shallow methods using hand-crafted features. For example, the recently proposed Copula Convolutional Neural Network (CNN) [11] for AU Intensity Estimation is highly-related to our approach, because it combines a CNN with a probabilistic graphical model similar to the one employed in MI-DORF. Even though the Copula CNN requires intensity labels for all the frames during training, the reported results on the DISFA dataset are comparable to those achieved by our method.…”

Deep Structure Inference Network for Facial Action Unit Recognition

“…Two main ways of solving multi-label learning in AU recognition are either capturing correlations through fully-connected layers [31,8,15] or inferring structure through probabilistic graphical models (PGM) [30,24,7]. While the former can capture correlations between classes, this is not done explicitly.…”

“…Together with patch-learning, Zhao et al [30] used positive and negative competitions among AUs to model a discriminative multi-label classifier. Walecki et al [24] placed a CRF on top of deep representations learned by a CNN. Both components are trained iteratively to estimate AU intensity.…”

“…JPML [30] treats the problem by including pre-learned priors about AU correlations into their learning. Learning structured outputs has been studied by [30,24,7] using Graphical Models, mathematical models able to capture complex probabilistic high order inter-dependencies. However, these models are not end-to-end trainable.…”