EmotioNet Challenge: Recognition of facial expressions of emotion in the wild

Abstract

This paper details the methodology and results of the EmotioNet challenge. This challenge is the first to test the ability of computer vision algorithms in the automatic analysis of a large number of images of facial expressions of emotion in the wild. The challenge was divided into two tracks. The first track tested the ability of current computer vision algorithms in the automatic detection of action units (AUs). Specifically, we tested the detection of 11 AUs. The second track tested the algorithms' ability to recognize emotion categories in images of facial expressions. Specifically, we tested the recognition of 16 basic and compound emotion categories. The results of the challenge suggest that current computer vision and machine learning algorithms are unable to reliably solve these two tasks. The limitations of current algorithms are more apparent when trying to recognize emotion. We also show that current algorithms are not affected by mild resolution changes, small occluders, gender or age, but that 3D pose is a major limiting factor on performance. We provide an in-depth discussion of the points that need special attention moving forward.

Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

“…Best results were also obtained when the network was trained with the two coupling losses. It can be observed that this approach outperformed by 5.7% and 8.6% in F1 score and Unweighted Average Recall (UAR), respectively, the state-of-the-art and winner of EmotioNet Challenge, NTechLab's [36] approach, which used the Emotionet's images with compound annotation.…”

“…We compared these networks' performance with the performance of FaceBehaviorNet when trained with and without the TABLE 3: Performance evaluation of valence-arousal, seven basic expression and action units predictions on all used databases provided by the FaceBehaviorNet when trained with/without the coupling losses, under the two task relatedness scenarios; 'AFA Score' is the average between the F1 Score and the Accuracy coupling losses. We also compared them with the performance of the state-of-the-art (sota) methodologies for each utilized database: i) the best performing CNN (VGG-FACE) [47] [9] on Aff-Wild; ii) the best performing network (AffWildNet) [47] [9] on Aff-Wild; iii) the baseline networks (AlexNet) [11] on AffectNet (in Table 5 they are denoted as '(2 ×) AlexNet' as they are two different networks: one for VA estimation and another for expression classification); iv) the state-of-the-art VGG-FACE [48] for VA estimation on AffectNet; v) the state-of-the-art RAN-ResNet18 + [49] for expression classification on AffectNet; vi) the VGG-FACE-mSVM [34] on RAF-DB; vii) the best performing network (DLP-CNN) [34] on RAF-DB; viii) the baseline network (AlexNet) [36] on EmotioNet ; ix) the winner of EmotioNet Challenge and best performing network (ResNet-34) [50] on Emo-tioNet; x) the state-of-the-art network (LP-Net) [52] on DISFA; xi) the best performing network (LP-Net) [52] on DISFA; xii) the winner of FERA 2015, DLE extension [53] on BP4D; xiii) the winner of FERA 2017 (VGG-FACE) [54] on BP4D+; xiv) the best performing network (ARL) [55] on BP4D+. Table 5 displays the performance of all these networks.…”

“…Let us first describe the databases that we utilized in all our experiments. In the affective computing case study, we selected to The EmotioNet database [35] contains around 1M images and was released for the EmotioNet Challenge in 2017 [36]. 950K images were automatically annotated and the remaining 50K images were manually annotated with 11 AUs (1,2,4,5,6,9,12,17,20,25,26); around half of the latter constituted the validation and the other half the test set of the Challenge.…”

Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework

“…For the past twenty years research in automatic analysis of facial behavior was mainly limited to posed behaviors which were captured in highly controlled recording conditions. Nevertheless, it is now accepted by the community [13] dynamic 12 action units controlled 54 videos: 261,630 frames BP4DS [14] dynamic 27 action units controlled 1,640 videos: 222,573 frames BP4D+ [15] dynamic 34 action units controlled 5,463 videos: 967,570 frames EmotioNet [18] static 11 action units in-the-wild 50,000 manual & 950,000 automatic annotations AFEW-VA [21] dynamic valence-arousal in-the-wild 600 videos: 30,050 frames SEWA [19] A/V valence-arousal in-the-wild 538 videos OMG-Emotion [20] A/V valence-arousal in-the-wild 495 videos: 5,288 utterances that facial expressions of naturalistic behaviors can be radically different from posed ones. Hence, efforts have been made in order to collect subjects displaying naturalistic behavior.…”

How to Synthesize a Large-Scale and Trainable Micro-Expression Dataset?

“…Our core idea is straightforward: it is easy to obtain a large number of faces from identity-labeled face datasets (e.g. LFW [14], EmotionNet [4]), and we can employ a conditional generative model to "add" MiEs onto these faces. The conditional facial expression generative model is a well-studied technique, and we adopt an off-theshelf algorithm, GANimation [31], which employs coefficients of Action Units (AUs) as the generative conditions.…”