Confidence-Weighted Local Expression Predictions for Occlusion Handling in Expression Recognition and Action Unit Detection

Abstract

Fully-Automatic Facial Expression Recognition (FER) from still images is a challenging task as it involves handling large interpersonal morphological differences, and as partial occlusions can occasionally happen. Furthermore, labelling expressions is a time-consuming process that is prone to subjectivity, thus the variability may not be fully covered by the training data. In this work, we propose to train Random Forests upon spatially defined local subspaces of the face. The output local predictions form a categorical expression-driven high-level representation that we call Local Expression Predictions (LEPs). LEPs can be combined to describe categorical facial expressions as well as Action Units (AUs). Furthermore, LEPs can be weighted by confidence scores provided by an autoencoder network. Such network is trained to locally capture the manifold of the non-occluded training data in a hierarchical way. Extensive experiments show that the proposed LEP representation yields high descriptive power for categorical expressions and AU occurrence prediction, and leads to interesting perspectives towards the design of occlusion-robust and confidence-aware FER systems.

Children with autism spectrum disorder produce more ambiguous and less socially meaningful facial expressions: an experimental study using random forest classifiers

“…We also aimed to characterize FE production in children with ASD with an FE recognition algorithm. Based on previous works in computer vision [44], we successfully combined facial landmarks and feature extraction with a random forest classifier. The accuracy reached when testing FEs of TD children after training on TD children were very good to excellent, except for sadness, which is a FE that is still difficult to produce in children in an experimental context.…”

Facial expressions analysis under occlusions based on specificities of facial motion propagation

“…However, these approaches require large dictionaries covering variations for each expression in order to build accurate linear combinations and in order to have enough characteristics to discriminate between expressions. In the sub-regions approaches, the face is divided into di↵erent regions and each region is analyzed individually [7]. The results are then merged to recognize the expression.…”

Facial Expression Analysis under Partial Occlusion

“…The approach achieved more than 87% accuracy for occlusion of the mouth, eyes, left and right sides of the face using JAFFE images. Dapogny et al [2016] presented Local Expression Predictions (LEPs) for categorical FER and AU prediction under partial occlusions. The LEPs were generated by locally averaging predictions by local trees in random forests which are trained using random facial masks generated in specific parts of the face.…”