Learning facial action units with spatiotemporal cues and multi-label sampling

Abstract

Facial action units (AUs) may be represented spatially, temporally, and in terms of their correlation. Previous research focuses on one or another of these aspects or addresses them disjointly. We propose a hybrid network architecture that jointly models spatial and temporal representations and their correlation. In particular, we use a Convolutional Neural Network (CNN) to learn spatial representations, and a Long Short-Term Memory (LSTM) to model temporal dependencies among them. The outputs of CNNs and LSTMs are aggregated into a fusion network to produce per-frame prediction of multiple AUs. The hybrid network was compared to previous state-of-the-art approaches in two large FACS-coded video databases, GFT and BP4D, with over 400,000 AU-coded frames of spontaneous facial behavior in varied social contexts. Relative to standard multi-label CNN and feature-based state-of-the-art approaches, the hybrid system reduced person-specific biases and obtained increased accuracy for AU detection. To address class imbalance within and between batches during training the network, we introduce multi-labeling sampling strategies that further increase accuracy when AUs are relatively sparse. Finally, we provide visualization of the learned AU models, which, to the best of our best knowledge, reveal for the first time how machines see AUs.

Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future

“…Early FEA work often included a computationally intensive and laborious process (e.g., face and facial landmark detection, hand-crafted feature extraction, and limited classification methods). Nowadays, researchers benefit from having access to comprehensive, large-scale facial datasets, as well as advanced computing resources to develop more efficient facial analysis methods [68,84,85,110,118,120].…”

Systematic Evaluation of Design Choices for Deep Facial Action Coding Across Pose

“…Recently, multi-label stratified sampling was found advantageous over naive sampling strategies for AU detection (Chu et al, 2019). In this experiment, we employed this strategy and investigated the effect of different training set sizes on the performance.…”

Spatiotemporal Features and Local Relationship Learning for Facial Action Unit Intensity Regression

“…Li et al [11] predicted AUs through multi-label learning and optimal temporal fusion using a long short-term memory (LSTM) network. Chu et al [12] proposed a structure comprising a 2D CNN and a LSTM network, where the 2D CNN learns spatial representations, and the LSTM models learn their temporal dependencies.…”

“…In this way, l/32 × l/32 local features can be obtained, and through local relationship learning with a bidirectional LSTM (BLSTM) structure, we can obtain a feature vector, which is fed into fully connected layers to regress the AU intensity. Considering methods like [11] and [12], and the results of pre-experiments, the depth of the BLSTM network is set to 2.…”