Fusing Multilabel Deep Networks for Facial Action Unit Detection

Abstract

The automatic detection of the activation of facial muscles, i.e. the detection of the so called facial Action Units (AUs), has received significant attention due to the application of facial expression analysis/recognition in areas such as affect recognition or behavior analysis. However, the recognition of subtle expressions is a challenging task that requires a multimodal approach where several sources of information are used. In this paper, we follow such an approach and propose a novel Deep Learning architecture that fuses information from several specialized Deep Neural Networks (DNNs) each of which models a different aspect of the problem in question. At the core of our approach is a novel dynamic adaptation of the Deep Network cost function so as to deal with the data imbalances that are inherent in multilabel classification problems-this allows crossdatabase training. We show the benefits of the proposed training approach and how different architectures are more suitable for particular AUs. Extensive experimental results show that our multi-modal approach outperform the state of the art by a considerable margin.

SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis

“…In Table 1, we also show the performance of Static [18] on the testing splits -we observe that our method obtains better results in the 8 expressions. This considerable difference in performance is mainly due to two reasons.…”

“…detection of the activation of certain facial muscles, is recently being treated as a pattern recognition problem, where one trains in a supervised manner classifiers that receive as input an image, or features extracted from it, and give at the output a set of binary labels, as many as the AUs that the method detects. In recent years the low-level feature extraction and the classifiers are replaced by Convolutional Neural Networks (CNNs) [18], [35], [36] since they have been shown to learn better and more general appearance features compared to hand-crafted ones [37]. Furthermore, networks trained on large datasets on surrogate tasks (e.g.…”

“…State-of-the-Art. We compare our AFEA method with Bishay and Patras [18], a method that achieves state-of-the-art results on the BP4D dataset [49] for facial expression recognition. The comparison is two-fold.…”

“…We observe that the performance is highly dependent on the number of training samples and the variance in expression-appearance. More specifically, expressions like lips part, and eyes closed have TABLE 1 The classification results obtained by the frame-based components of the network proposed in [18] and by the proposed AFEA method over different facial expressions on the 15% testing splits of the EmotioNet [43], ExpW [44], CelebA [45], and CEW [46] On other expressions like brow lowerer, and lid tightener we obtain moderately good performance due to the large variance in expression-appearance among different people. Finally, we obtain low F1-score values for the outer brow raiser and the upper lid raiser as the EmotioNet dataset is highly imbalanced for those two classes.…”

“…The proposed network has been trained in a personindependent manner to predict expression-related symptoms from two commonly-used assessment interviews; Positive and Negative Syndrome Scale (PANSS) [16], and Clinical Assessment Interview for Negative Symptoms (CAINS) [17]. Experimental results show that training the Facial Expression Analysis sub-network (stage 1) "in the wild" delivers better performance on symptom severity estimation in comparison to another state-of-the-art method [18] that is trained using data captured in a controlled environment. Furthermore, we show that high and statistically significant correlations between the detected expressions and the severity of several symptoms in both the PANSS and CAINS can be obtained.…”