Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model

Do, Nhu-Tai; Nguyen-Quynh, Tram-Tran; Kim, Soo-Hyung

doi:10.1109/fg47880.2020.00093

Cited by 15 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the one hand, self-reporting emotions and objectively measured emotions are essentially different emotional attributes. On the other hand, we can interpret such differences as an inability of objective assessment to recognize all subjective characteristics of the emotions expressed on faces [40]. However, the method of self-reporting is simple.…”

Section: The Differences Between the 2 Emotional Measurement Methodsmentioning

confidence: 99%

“…Then, we captured the facial movements of the participants with camera videos. To recognize facial expressions of the video we captured, we used the face recognition model proposed by Do et al, 2020 [40]. After we input the videos to the model, the valence-arousal emotion values were output for the subsequent analysis.…”

Section: Physiological Measuresmentioning

confidence: 99%

“…However, emotion recognition remains a complex problem [35]. The use of neural networks allows us to exceed the accuracy of manual identification [40], but each additional percentage of accuracy is achieved through complex scientific research and experiments [41].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Assessing Emotional Responses to the Spatial Quality of Urban Green Spaces through Self-Report and Face Recognition Measures

Lin

Zhuang

Zhang

et al. 2021

IJERPH

View full text Add to dashboard Cite

Although creating a high-quality urban green space (UGS) is of considerable importance in public health, few studies have used individuals’ emotions to evaluate the UGS quality. This study aims to conduct a multidimensional emotional assessment method of UGS from the perspective of spatial quality. Panoramic videos of 15 scenes in the West Lake Scenic Area were displayed to 34 participants. For each scene, 12 attributes regarding spatial quality were quantified, including perceived plant attributes, spatial structure attributes, and experiences of UGS. Then, the Self-Assessment-Manikin (SAM) scale and face recognition model were used to measure people’s valence-arousal emotion values. Among all the predictors, the percentages of water and plants were the most predictive indicators of emotional responses measured by SAM scale, while the interpretation rate of the model measured by face recognition was insufficiently high. Concerning gender differences, women experienced a significantly higher valence than men. Higher percentages of water and plants, larger sizes, approximate shape index, and lower canopy densities were often related to positive emotions. Hence, designers must consider all structural attributes of green spaces, as well as enrich visual perception and provide various activities while creating a UGS. In addition, we suggest combining both physiological and psychological methods to assess emotional responses in future studies. Because the face recognition model can provide objective measurement of emotional responses, and the self-report questionnaire is much easier to administer and can be used as a supplement.

show abstract

Section: The Differences Between the 2 Emotional Measurement Methodsmentioning

confidence: 99%

Section: Physiological Measuresmentioning

confidence: 99%

See 1 more Smart Citation

Assessing Emotional Responses to the Spatial Quality of Urban Green Spaces through Self-Report and Face Recognition Measures

Lin

Zhuang

Zhang

et al. 2021

IJERPH

View full text Add to dashboard Cite

show abstract

“…Several multi-task learning models [7,9,26] effectively leveraged the availability of Aff-wild2 data jointly annotated with the labels of dimensional affect, categorical expressions, and AUs. A holistic multi-task, multi-domain network for facial emotion analysis named FaceBehavior-Net was developed on Aff-wild2 and validated in a crosscorpus setting in [19,20,23].…”

Section: Related Workmentioning

confidence: 99%

Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition

Karas¹,

Tellamekala²,

Mallol-Ragolta³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present our submission to 3rd Affective Behavior Analysis in-the-wild (ABAW) challenge. Learning complex interactions among multimodal sequences is critical to recognise dimensional affect from in-the-wild audiovisual data. Recurrence and attention are the two widely used sequence modelling mechanisms in the literature. To clearly understand the performance differences between recurrent and attention models in audiovisual affect recognition, we present a comprehensive evaluation of fusion models based on LSTM-RNNs, self-attention and cross-modal attention, trained for valence and arousal estimation. Particularly, we study the impact of some key design choices: the modelling complexity of CNN backbones that provide features to the the temporal models, with and without endto-end learning. We trained the audiovisual affect recognition models on in-the-wild ABAW corpus by systematically tuning the hyper-parameters involved in the network architecture design and training optimisation. Our extensive evaluation of the audiovisual fusion models shows that LSTM-RNNs can outperform the attention models when coupled with low-complex CNN backbones and trained in an end-to-end fashion, implying that attention models may not necessarily be the optimal choice for continuous-time multimodal emotion recognition.

show abstract

“…The third ABAW Competition, to be held in conjunction with the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022 is a continuation of the first [24] and second [32] ABAW Competitions held in conjunction with the IEEE Conference on Face and Gesture Recognition (IEEE FG) 2021 and with the International Conference on Computer Vision (ICCV) 2022, respectively, which targeted dimensional (in terms of valence and arousal) [2][3][4]8,9,11,21,35,39,47,48,50,[54][55][56], categorical (in terms of the basic expressions) [12,15,16,33,36,37,51] and facial action unit analysis and recognition [7,19,20,25,26,40,44,47]. The third ABAW Competition contains four Challenges, which are based on the same in-the-wild database, (i) the uni-task Valence-Arousal Estimation Challenge; (ii) the uni-task Expression Classification Challenge (for the 6 basic expressions plus the neutral state plus the 'other' category that denotes expressions/affective states other than the 6 basic ones); (iii) the uni-task Action Unit Detection Challenge (for 12 action units); (iv) the Multi-Task Learning Challenge (for joint learning and predicting of valence-arousal, 8 expressions -6 basic plus neutral plus 'other'-and 12 action units).…”

Section: Introductionmentioning

confidence: 99%

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges

Kollias¹

2022

Preprint

View full text Add to dashboard Cite

This paper describes the third Affective Behavior Analysis in-the-wild (ABAW) Competition, held in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. The 3rd ABAW Competition is a continuation of the Competitions held at ICCV 2021, IEEE FG 2020 and IEEE CVPR 2017 Conferences, and aims at automatically analyzing affect. This year the Competition encompasses four Challenges: i) uni-taskValence-Arousal Estimation, ii) uni-task Expression Classification, iii) uni-task Action Unit Detection, and iv) Multi-Task-Learning. All the Challenges are based on a common benchmark database, Aff-Wild2, which is a large scale in-the-wild database and the first one to be annotated in terms of valence-arousal, expressions and action units. In this paper, we present the four Challenges, with the utilized Competition corpora, we outline the evaluation metrics and present the baseline systems along with their obtained results.

show abstract

Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model

Cited by 15 publications

References 8 publications

Assessing Emotional Responses to the Spatial Quality of Urban Green Spaces through Self-Report and Face Recognition Measures

Assessing Emotional Responses to the Spatial Quality of Urban Green Spaces through Self-Report and Face Recognition Measures

Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges

Contact Info

Product

Resources

About