EAC-Net: Deep Nets with Enhancing and Cropping for Facial Action Unit Detection

Abstract

In this paper, we propose a deep learning based approach for facial action unit (AU) detection by enhancing and cropping regions of interest of face images. The approach is implemented by adding two novel nets (a.k.a. layers): the enhancing layers and the cropping layers, to a pretrained convolutional neural network (CNN) model. For the enhancing layers (noted as E-Net), we have designed an attention map based on facial landmark features and apply it to a pretrained neural network to conduct enhanced learning. For the cropping layers (noted as C-Net ), we crop facial regions around the detected landmarks and design individual convolutional layers to learn deeper features for each facial region. We then combine the E-Net and the C-Net to construct a so-called Enhancing and Cropping Net (EAC-Net), which can learn both features enhancing and region cropping functions effectively. The EAC-Net integrates three important elements, i.e., learning transfer, attention coding, and regions of interest processing, making our AU detection approach more efficient and more robust to facial position and orientation changes. Our approach shows a significant performance improvement over the state-of-the-art methods when tested on the BP4D and DISFA AU datasets. The EAC-Net with a slight modification also shows its potentials in estimating accurate AU intensities. We have also studied the performance of the proposed EAC-Net under two very challenging conditions: (1) faces with partial occlusion and (2) faces with large head pose variations. Experimental results show that (1) the EAC-Net learns facial AUs correlation effectively and predicts AUs reliably even with only half of a face being visible, especially for the lower half; (2) Our EAC-Net model also works well under very large head poses, which outperforms significantly a compared baseline approach. It further shows that the EAC-Net works much better without a face frontalization than with face frontalization through image warping as pre-processing, in terms of computational efficiency and AU detection accuracy.

D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection

“…Jaiswal and Valstar (2016) use a pre-defined binary mask created to select a relevant region for a particular AU and pass it to a convolutional and bidirectional Long Short-Term Memory (LSTM) neural network. Li et al (2018) design an attention map using the facial key points and AU centers to enforce their CNN-based architecture to focus more on these AU centers. Sanchez et al (2018) generate heatmaps for a target AU, by estimating the facial landmarks and drawing a 2D Gaussian around the points where the AU is known to cause changes.…”

“…Adversarial Training Framework (ATF) (Zhang et al, 2018) is a CNN-based framework in which AU loss is minimized and identity loss is maximized to learn subject invariant feature representations during the adversarial training. Finetuned VGG Network (FVGG) (Li et al, 2018) is the model obtained after finetuning the pretrained VGG 19-layer model. Network with enhancing layers (E-Net) (Li et al, 2018) is the finetuned VGG network with enhancing layer which forces the network to pay more attention to AU interest regions on face images.…”

“…Finetuned VGG Network (FVGG) (Li et al, 2018) is the model obtained after finetuning the pretrained VGG 19-layer model. Network with enhancing layers (E-Net) (Li et al, 2018) is the finetuned VGG network with enhancing layer which forces the network to pay more attention to AU interest regions on face images. Enhancing and Cropping Network (EAC Net) (Li et al, 2018) is a pretrained CNN model with enhancing (E-Net) and cropping (C-Net) layers.…”

Guest Editorial: The Computational Face

“…The contributions focused on a diverse set of topics, techniques, and applications. Estimation of facial markers (e.g., facial landmarks, action units) was the most approached task in papers within the special issue [4], [5], [6], [7]. In fact, most papers in the issue used facial landmarks in one way or another as part of their methodology.…”

“…A similar methodology was adopted by Wang et al who combined a super resolution CNN and a recurrent neural net to iteratively refine landmarks in a cascade regression approach [6]. A novel deep architecture was proposed by Li et al in which attention maps and cropping layers were included in a CNN for better detection of facial action units [7]. In contrast to previous work, Wang et al described an iterative methodology based on deformable parts models for person-specific landmark detection.…”