Systematic Evaluation of Design Choices for Deep Facial Action Coding Across Pose

Abstract

The performance of automated facial expression coding is improving steadily. Advances in deep learning techniques have been key to this success. While the advantage of modern deep learning techniques is clear, the contribution of critical design choices remains largely unknown, especially for facial action unit occurrence and intensity across pose. Using the The Facial Expression Recognition and Analysis 2017 (FERA 2017) database, which provides a common protocol to evaluate robustness to pose variation, we systematically evaluated design choices in pre-training, feature alignment, model size selection, and optimizer details. Informed by the findings, we developed an architecture that exceeds state-of-the-art on FERA 2017. The architecture achieved a 3.5% increase in F1 score for occurrence detection and a 5.8% increase in Intraclass Correlation (ICC) for intensity estimation. To evaluate the generalizability of the architecture to unseen poses and new dataset domains, we performed experiments across pose in FERA 2017 and across domains in Denver Intensity of Spontaneous Facial Action (DISFA) and the UNBC Pain Archive.

Viewpoint Robustness of Automated Facial Action Unit Detection Systems

“…When developing an algorithm, the design is informed by the robustness to angle changes. Niimura et al [55] evaluated robustness to pose variation, including pitch, of algorithms for "deep facial action coding" according to pre-training, feature alignment, model size, and optimizers. Many computer vision researchers have attempted to develop pose normalization techniques [56,57].…”

Facial Expression Manipulation for Personalized Facial Action Estimation

“…Then we synthesized 5,000 images for these pairs. We selected 5,000 images according to experimental results by Niinuma et al (2021b). They analyzed the influence of training set size on FERA17, and showed that the training set size have a minor influence on the performance: score peaked at 5,000 images, and after that performance plateaued.…”

“…We selected a VGG16 network pre-trained on ImageNet for the baseline architecture for AU estimation. Previous studies found this combination preferable for AU coding (Niinuma et al, 2021b). We replaced the final layer of the network with a 6-length one-hot representation, and fine-tuned VGG16 network from the third convolutional layer.…”

“…We replaced the final layer of the network with a 6-length one-hot representation, and fine-tuned VGG16 network from the third convolutional layer. The dropout rate was set to 0.5, and an Adam optimizer was used with LR = 5 × 10 -5 as suggested in (Niinuma et al, 2021b). We fine-tuned the pre-trained model for 10 epochs, chose the model showing the best performance on the validation partition from the 10 models, and then reported the results on the subject-independent test partition.…”