Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze

Wu, Suowei; Du, Zhengyin; Li, Weixin; Huang, Di; Wang, Yunlong

doi:10.1145/3340555.3353739

Cited by 28 publications

(15 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While there has been research on fine-grained, temporally precise emotion recognition (cf., FEELtrace [ 17 ], DARMA [ 18 ], CASE [ 19 ]), these methods either require users to wear or attach obtrusive sensors [ 20 , 21 , 22 ] (e.g., Electroencephalograph (EEG)), or rely on facial expression sensing [ 20 , 21 , 23 , 24 ] for fine-grained emotion recognition. With respect to EEG, emotion recognition accuracies up to

have been achieved over the past decade [ 25 ].…”

Section: Introductionmentioning

confidence: 99%

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Zhang

Ali

Wang³

et al. 2020

Sensors

View full text Add to dashboard Cite

Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using only wearable, physiological signals (e.g., electrodermal activity, heart rate). CorrNet takes advantage of features both inside each instance (intra-modality features) and between different instances for the same video stimuli (correlation-based features). We first test our approach on an indoor-desktop affect dataset (CASE), and thereafter on an outdoor-mobile affect dataset (MERCA) which we collected using a smart wristband and wearable eyetracker. Results show that for subject-independent binary classification (high-low), CorrNet yields promising recognition accuracies: 76.37% and 74.03% for V-A on CASE, and 70.29% and 68.15% for V-A on MERCA. Our findings show: (1) instance segment lengths between 1–4 s result in highest recognition accuracies (2) accuracies between laboratory-grade and wearable sensors are comparable, even under low sampling rates (≤64 Hz) (3) large amounts of neutral V-A labels, an artifact of continuous affect annotation, result in varied recognition performance.

show abstract

have been achieved over the past decade [ 25 ].…”

Section: Introductionmentioning

confidence: 99%

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Zhang

Ali

Wang³

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Attentiveness (gaze) is also a crucial part of affect analysis, especially for people with communication disabilities [10]. Recent works showed a strong link between emotion recognition and eye gaze estimation [41], [42] which motivates us to use gaze to measure temporal expressiveness dynamics. Thus, we incorporate eye-gaze features in our algorithm by capturing gaze from the left eye and right eye.…”

Section: Methods a Motivation And Multimodal Facial Features Trackingmentioning

confidence: 99%

Quantified Facial Temporal-Expressiveness Dynamics for Affect Analysis

Uddin¹,

Canavan²

2020

Preprint

View full text Add to dashboard Cite

The quantification of visual affect data (e.g. face images) is essential to build and monitor automated affect modeling systems efficiently. Considering this, this work proposes quantified facial Temporal-expressiveness Dynamics (TED) to quantify the expressiveness of human faces. The proposed algorithm leverages multimodal facial features by incorporating static and dynamic information to enable accurate measurements of facial expressiveness. We show that TED can be used for high-level tasks such as summarization of unstructured visual data, and expectation from and interpretation of automated affect recognition models. To evaluate the positive impact of using TED, a case study was conducted on spontaneous pain using the UNBC-McMaster spontaneous shoulder pain dataset. Experimental results show the efficacy of using TED for quantified affect analysis.

show abstract

“…Then, they fed the pose into a classifier to obtain the emotion. Pupil size [ 47 ] and gaze [ 48 ] are also important features used for recognizing emotion.…”

Section: Related Workmentioning

confidence: 99%

Incorporating Interpersonal Synchronization Features for Automatic Emotion Recognition from Visual and Audio Data during Communication

Quan

Miyake

Nozawa

2021

Sensors

View full text Add to dashboard Cite

During social interaction, humans recognize others’ emotions via individual features and interpersonal features. However, most previous automatic emotion recognition techniques only used individual features—they have not tested the importance of interpersonal features. In the present study, we asked whether interpersonal features, especially time-lagged synchronization features, are beneficial to the performance of automatic emotion recognition techniques. We explored this question in the main experiment (speaker-dependent emotion recognition) and supplementary experiment (speaker-independent emotion recognition) by building an individual framework and interpersonal framework in visual, audio, and cross-modality, respectively. Our main experiment results showed that the interpersonal framework outperformed the individual framework in every modality. Our supplementary experiment showed—even for unknown communication pairs—that the interpersonal framework led to a better performance. Therefore, we concluded that interpersonal features are useful to boost the performance of automatic emotion recognition tasks. We hope to raise attention to interpersonal features in this study.

show abstract

Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze

Cited by 28 publications

References 35 publications

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Quantified Facial Temporal-Expressiveness Dynamics for Affect Analysis

Incorporating Interpersonal Synchronization Features for Automatic Emotion Recognition from Visual and Audio Data during Communication

Contact Info

Product

Resources

About