An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild

Abstract

In this work we tackle the task of video-based audiovisual emotion recognition, within the premises of the 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). Poor illumination conditions, head/body orientation and low image resolution constitute factors that can potentially hinder performance in case of methodologies that solely rely on the extraction and analysis of facial features. In order to alleviate this problem, we leverage bodily as well as contextual features, as part of a broader emotion recognition framework. We choose to use a standard CNN-RNN cascade as the backbone of our proposed model for sequence-to-sequence (seq2seq) learning. Apart from learning through the RGB input modality, we construct an aural stream which operates on sequences of extracted mel-spectrograms. Our extensive experiments on the challenging and newly assembled Affect-in-the-wild-2 (Aff-Wild2) dataset verify the superiority of our methods over existing approaches, while by properly incorporating all of the aforementioned modules in a network ensemble, we manage to surpass the previous best published recognition scores, in the official validation set. All the code was implemented using PyTorch 1 and is publicly available 2 . * equal contribution 1 https://pytorch.org/ 2 https://github.com/PanosAntoniadis/ NTUA-ABAW2021

Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training

“…Since, we have trained model for single task and not used any of audio and video features, so performance is not as good as teams using multi-task learning with video features. [27] 0.29 0.6491 0.4082 NTUA-CVSP [28] 0.3367 0.6418 0.4374 Morphoboid [29] 0.3511 0.668 0.4556 FLAB2021 [30] 0.4079 0.6729 0.4953 STAR [31] 0.4759 0.7321 0.5604 Maybe Next Time [32] 0.6046 0.7289 0.6456 CPIC-DIR2021 [33] 0.6834 0.7709 0.7123 Netease Fuxi Virtual Human [34] 0.763 0.8059 0.7777 Ours [18] 0.361 0.675 0.4646 Table 3 shows the influence of number of networks that are collaboratively trained in CCT. It can be observed that model with 3 networks performs the best in the presence of noise.…”

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges

“…The third ABAW Competition, to be held in conjunction with the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022 is a continuation of the first [24] and second [32] ABAW Competitions held in conjunction with the IEEE Conference on Face and Gesture Recognition (IEEE FG) 2021 and with the International Conference on Computer Vision (ICCV) 2022, respectively, which targeted dimensional (in terms of valence and arousal) [2][3][4]8,9,11,21,35,39,47,48,50,[54][55][56], categorical (in terms of the basic expressions) [12,15,16,33,36,37,51] and facial action unit analysis and recognition [7,19,20,25,26,40,44,47]. The third ABAW Competition contains four Challenges, which are based on the same in-the-wild database, (i) the uni-task Valence-Arousal Estimation Challenge; (ii) the uni-task Expression Classification Challenge (for the 6 basic expressions plus the neutral state plus the 'other' category that denotes expressions/affective states other than the 6 basic ones); (iii) the uni-task Action Unit Detection Challenge (for 12 action units); (iv) the Multi-Task Learning Challenge (for joint learning and predicting of valence-arousal, 8 expressions -6 basic plus neutral plus 'other'-and 12 action units).…”