EAC-Net: A Region-Based Deep Enhancing and Cropping Approach for Facial Action Unit Detection

Abstract

In this paper, we propose a deep learning based approach for facial action unit detection by enhancing and cropping the regions of interest. The approach is implemented by adding two novel nets (layers): the enhancing layers and the cropping layers, to a pretrained CNN model. For the enhancing layers, we designed an attention map based on facial landmark features and applied it to a pretrained neural network to conduct enhanced learning (The E-Net). For the cropping layers, we crop facial regions around the detected landmarks and design convolutional layers to learn deeper features for each facial region (C-Net). We then fuse the E-Net and the C-Net to obtain our Enhancing and Cropping (EAC) Net, which can learn both feature enhancing and region cropping functions. Our approach shows significant improvement in performance compared to the state-of-the-art methods applied to BP4D and DISFA AU datasets.

Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection

“…We compare our method AU-GCN against state-of-the-art single-image based AU detection works under the same 3-fold cross validation setting. These methods include both traditional methods, LSVM [6], JPML [25], and deep learning methods, LCN [21], DRML [26] and EAC-Net [11]. Note that EAC-Net [11] is not compared AUC due to its metrics of accuracy instead of AUC.…”

“…Although this approach treats each face as a group of individual parts, it divides blocks uniformly and does not consider the FACS knowledge, thereby leading to poor performance. Wei Li et al [11] proposed Enhancing and Cropping Net (EAC-Net), which intends to give significant attention to individual AU centers; however, this approach does not consider AU relationship modeling, and the lack of RoI-level supervised information can only give coarse guidance. All these researches demonstrate the effectiveness of deep learning on feature extraction for AU detection task.…”

“…For an image with the relatively fixed structure, such as a human face, a standard CNN may fail to capture those subtle appearance changes. In order to focus more on AU specific regions, the AU local region partition rules are defined at first by taking FACS [5] and EAC-Net [11] as reference.…”

“…Dataset: The effectiveness of our proposed AU-GCN is evaluated on two benchmark datasets: BP4D [24] and DISFA [16]. For BP4D and DISFA, a 3-fold partition is adopted to ensure subjects were mutually exclusive in train/val/test sets by following previous related work [26,11]. The frames with intensities equal or greater than 2 are considered as positive, while others are treated as negative.…”

Region Based Adversarial Synthesis of Facial Action Units

“…It is a certain reference for AU synthesis that most works on AU detection make full use of landmark-based geometry to improve the performance. Li et al [12] proposed a deep learning based approach named EAC-Net for facial action unit detection by enhancing and cropping the AU regions of interest(ROI) with roughly extracted facial landmark information.Therefore, in this paper, imitating EAC-Net, our proposed framework separates the facial image into multiple local AU regions which are integrated in modern deep learning model.In addition, attention-GAN [1] proposed adopts attention mechanism to focus on generating objects of interests without touching the background region. Simi-lar to them, we proposed AU ROI localization module to maintain the other information except the concerned AU.…”