Facial Action Units-Based Image Retrieval for Facial Expression Recognition

Abstract

Facial expression recognition (FER) is a very challenging problem in computer vision. Although extensive research has been conducted to improve FER performance in recent years, there is still room for improvement. A common goal of FER is to classify a given face image into one of seven emotion categories: angry, disgust, fear, happy, neutral, sad, and surprise. In this paper, we propose to use a simple multi-layer perceptron (MLP) classifier that determines whether the current classification result is reliable or not. If the current classification result is determined as unreliable, we use the given face image as a query to search for similar images. In particular, facial action units are used to retrieve the images with a similar facial expression. Then, another MLP is trained to predict final emotion category by aggregating classification output vectors of the query image and its retrieved similar images. Experimental results on FER2013 dataset demonstrate that the performance of the state-of-the-art networks can be further improved by our proposed method. INDEX TERMSConvolutional neural networks, facial expression recognition, image retrieval, facial action units.

Predicting Depression, Anxiety, and Stress Levels from Videos Using the Facial Action Coding System

“…All of these selected AUs are part of the standard set used in most facial expression recognition (FER) systems based on FACS [44,45]. We excluded AU41 (lid droop), AU42 (slit), and AU46 (wink) from the upper part of the face because these AUs were not coded with intensity levels in the last FACS revision [43], and a binary classification (AU present/absent) in a context where all other AUs were evaluated based on a level from O to E could negatively impact the accuracy of the system.…”

“…Therefore, we used 31 AUs that could be classified fast and with high accuracy: For the remaining AUs, more complex methods would have been required, which would also have required more processing time, so they were not evaluated in this study, since we aimed to build a system able to predict the DASS levels in real time. Another reason for which we selected this set of AUs was that they are present in several large databases employed for testing FER systems [44,45], such as Cohn-Kanade extended (CK+) and MMI, which we used in this study for evaluating the accuracy of the AU classification task. All AUs analyzed were treated individually, as nonadditive.…”

Facial Action Units for Training Convolutional Neural Networks

“…In [37], the AU-based image retrieval method was used to get supplementary information on the classification result of the given test image. Specifically, the retrieved image as well as the original test image participate in making final classification.…”

“…In this sub-section, we introduce an AU-based image retrieval method to find images with the similar emotion, not with the similar face, to increase the number of images in the minority classes. The AU-based image retrieval approach has been already used in [37], where the AU-based image retrieval was used in the testing stage to provide additional test images to increase the reliability on the classification score. In this paper, however, the AU-based image retrieval is adopted in the training stage to find the images with the similar emotion to the query image.…”

“…Then, the retrieved images are added to the minority classes in the training dataset to resolve the imbalance problem. Based on the OpenFace [38] implementation, the AU-based image retrieval in [37] detect the binary AU-features for 18 AUs and the Jaccard index [39] was used for the similarity measure. On the other hand, in this paper, we adopt 17 AU intensity values (i.e., not the binary ones as in [37]) ranging from 0 to 5, which are normalized in the range of 0 to 1 to be a 17-dimensinal feature vector for the similarity matching.…”