ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection &amp; Multi-Task Learning Challenges

Kollias, Dimitrios

doi:10.1109/cvprw56347.2022.00259

Cited by 86 publications

(57 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ResNet50 model was originally proposed for general image recognition tasks and later it was retrained with the VGGFace2 database [49] for face recognition. This architecture has been extensively used as a starting point in the Facial expressions analysis [57][58][59] and Action Units recognition in competitions like Affective Behavior Analysis in-thewild (ABAW) in FG 2020 [60], ICCV 2021 [61], and CVPR 2022 [62]. The architecture is used as feature extractor by removing the final decision layer.…”

Section: Plos Onementioning

confidence: 99%

Exploring facial expressions and action unit domains for Parkinson detection

et al. 2023

View full text Add to dashboard Cite

Background and objective Patients suffering from Parkinson’s disease (PD) present a reduction in facial movements called hypomimia. In this work, we propose to use machine learning facial expression analysis from face images based on action unit domains to improve PD detection. We propose different domain adaptation techniques to exploit the latest advances in automatic face analysis and face action unit detection. Methods Three different approaches are explored to model facial expressions of PD patients: (i) face analysis using single frame images and also using sequences of images, (ii) transfer learning from face analysis to action units recognition, and (iii) triplet-loss functions to improve the automatic classification between patients and healthy subjects. Results Real face images from PD patients show that it is possible to properly model elicited facial expressions using image sequences (neutral, onset-transition, apex, offset-transition, and neutral) with accuracy improvements of up to 5.5% (from 72.9% to 78.4%) with respect to single-image PD detection. We also show that our proposed action unit domain adaptation provides improvements of up to 8.9% (from 78.4% to 87.3%) with respect to face analysis. Finally, we also show that triplet-loss functions provide improvements of up to 3.6% (from 78.8% to 82.4%) with respect to action unit domain adaptation applied upon models created from scratch. The code of the experiments is available at https://github.com/luisf-gomez/Explorer-FE-AU-in-PD. Conclusions Domain adaptation via transfer learning methods seem to be a promising strategy to model hypomimia in PD patients. Considering the good results and also the fact that only up to five images per participant are considered in each sequence, we believe that this work is a step forward in the development of inexpensive computational systems suitable to model and quantify problems of PD patients in their facial expressions.

show abstract

Section: Plos Onementioning

confidence: 99%

Exploring facial expressions and action unit domains for Parkinson detection

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Some have been extended to many applications, including medical diagnosis [9], social media [10], e-education [11], and video generation [12]. Meanwhile, competitions such as FERA [13], EmotiW [14], Aff-Wild [15], ABAW [16], EmotioNet [17], AVEC [18], and MuSe [19] are regularly held to evaluate the latest progress and propose frontier research trends.…”

Section: Introductionmentioning

confidence: 99%

Graph-Based Facial Affect Analysis: A Review

Liu

Zhang

et al. 2023

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

As one of the most important affective signals, facial affect analysis (FAA) is essential for developing human-computer interaction systems. Early methods focus on extracting appearance and geometry features associated with human affects while ignoring the latent semantic information among individual facial changes, leading to limited performance and generalization. Recent work attempts to establish a graph-based representation to model these semantic relationships and develop frameworks to leverage them for various FAA tasks. This paper provides a comprehensive review of graph-based FAA, including the evolution of algorithms and their applications. First, the FAA background knowledge is introduced, especially on the role of the graph. We then discuss approaches widely used for graph-based affective representation in literature and show a trend towards graph construction. For the relational reasoning in graph-based FAA, existing studies are categorized according to their non-deep or deep learning methods, emphasizing the latest graph neural networks. Performance comparisons of the state-of-the-art graph-based FAA methods are also summarized. Finally, we discuss the challenges and potential directions. As far as we know, this is the first survey of graph-based FAA methods. Our findings can serve as a reference for future research in this field.

show abstract

“…From a machine learning perspective, AU detection in the wild presents many technical challenges. Most notably, in-the-wild datasets such as Aff-Wild2 [12][13][14][15][16][17][18][19][20][21]32] collect data with huge variations in the cameras (resulting in blurred video frames), environments (illumination conditions), and subjects (large variance in expressions, scale, and head poses). Ertugrul et al [4,5] demonstrate that the deep-learning-based AU detectors have limited generalization abilities due to the aforementioned variations.…”

Section: Introductionmentioning

confidence: 99%

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Yin¹,

Tran²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Facial action unit detection has emerged as an important task within facial expression analysis, aimed at detecting specific pre-defined, objective facial expressions, such as lip tightening and cheek raising. This paper presents our submission to the Affective Behavior Analysis in-the-wild (ABAW) 2023 Competition for AU detection. We propose a multi-modal method for facial action unit detection with visual, acoustic, and lexical features extracted from the large pre-trained models. To provide high-quality details for visual feature extraction, we apply super-resolution and face alignment to the training data. Our approach achieves the F1 score of 52.3% on the official validation set.

show abstract

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges

Cited by 86 publications

References 41 publications

Exploring facial expressions and action unit domains for Parkinson detection

Exploring facial expressions and action unit domains for Parkinson detection

Graph-Based Facial Affect Analysis: A Review

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Contact Info

Product

Resources

About