Facial expression analysis with AFFDEX and FACET: A validation study

Stöckli, Sabrina; Schulte‐Mecklenbeck, Michael; Borer, Stefan; Samson, Andrea Christiane

doi:10.3758/s13428-017-0996-1

Cited by 196 publications

(179 citation statements)

References 67 publications

Supporting

Mentioning

164

Contrasting

Unclassified

Order By: Relevance

“…We established the main effect of dataset type on recognition accuracy F(2, 7317) = 113.92, p < .001, η 2 p = .033, with AFEW containing the least detectable emotions (M = 0.40, 95% CI [0.38, 0.42], p < .001), and RAVDESS containing the most detectable emotions M = 0.57, 95% CI [0.56, 0.59], p < .001. Consistent with the results of other studies [3], accuracy scores for emotion labels were higher for acted facial expressions (RAVDESS M=0.57, 95% CI [0.56, 0.59]; SAVEE M = 0.54, 95% CI [0.52, 0.56])in comparison with more challenging 'in the wild' expressions (AFEW M =0.40, 95% CI [0.38, 0.42], p < .001). Significant interaction was revealed between algorithm type and dataset type F(6, 7317) = 7.45, p < .001, η 2 p = .007.…”

Section: Datasetsupporting

confidence: 93%

“…We tested the performance of four commercial algorithms on the datasets with different level of control over conditions of data acquisition. As different algorithms use distinct statistical methods and datasets to train the machine learning procedures, they differently classify emotions [3]. If algorithms are capable to recognize what people actually express (in other words, technologies work), then different algorithms would give consistent predictions per video recording, using particular emotion labels.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Overview of the Advancements in Automatic Emotion Recognition: Comparative Performance of Commercial Algorithms

Malygina¹,

Artemyev²,

Belyaev³

et al. 2019

Preprint

View full text Add to dashboard Cite

In the recent years facial emotion recognition algorithms have evolved and in some cases top commercial algorithms detect emotions like happiness better than humans do. To evaluate the performance of these algorithms, the common practice is to compare them with human-labeled ground truth. This article covers monitoring of the advancements in automatic emotion recognition solutions, and here we suggest an additional criteria for their evaluation, that is the agreement between algorithms' predictions. In this work, we compare the performance of four commercial algorithms: Affectiva Affdex, Microsoft Cognitive Services Face module Emotion Recognition, Amazon Rekognition Face Analysis, and Neurodata Lab Emotion Recognition on three datasets AFEW, RAVDESS, and SAVEE, that differ in terms of control over conditions of data acquisition. We assume that the consistency among algorithms' predictions indicates the reliability of the predicted emotion. Overall results show that the algorithms with higher accuracy and f1-scores that were obtained for human-labeled ground truth (Microsoft's, Neurodata Lab's, and Amazon's), showed higher agreement between their predictions. Agreement among algorithms' predictions is a promising criteria in terms of further exploring the option to replace human data labeling with automatic annotation.

show abstract

Section: Datasetsupporting

confidence: 93%

mentioning

confidence: 99%

Overview of the Advancements in Automatic Emotion Recognition: Comparative Performance of Commercial Algorithms

Malygina¹,

Artemyev²,

Belyaev³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…We then analyzed each of the 4,648 recordings with FACET (23). FACET is a computer-vision tool that automatically detects 20 FACS-based AUs (see Supplementary Table 1 for descriptions and depictions of FACET-detected AUs).…”

Section: Methodsmentioning

confidence: 99%

“…Instead, researchers tend to rely on measures of emotional responding that are not observable in social interactions (e.g., heart rate variability). Recently, automated computer-vision and machine learning (CVML) based approaches have emerged that make it possible to scale AU annotation to larger numbers of participants (e.g., 21-23) thus making follow-up studies more feasible. In fact, inter-disciplinary applications of CVML have allowed researchers to automatically identify pain severity (e.g., 24), depressive states (e.g., 25), and discrete emotions from facial expressions (e.g., 26).…”

Section: Using Computer-vision and Machine Learning To Automate Faciamentioning

confidence: 99%

Using Computer-vision and Machine Learning to Automate Facial Coding of Positive and Negative Affect Intensity

Haines

Southward

Cheavens

et al. 2018

Preprint

View full text Add to dashboard Cite

Facial expressions are fundamental to interpersonal communication, including social interaction, and allow people of different ages, cultures, and languages to quickly and reliably convey emotional information. Historically, facial expression research has followed from discrete emotion theories, which posit a limited number of distinct affective states that are represented with specific patterns of facial action. Much less work has focused on dimensional features of emotion, particularly positive and negative affect intensity. This is likely, in part, because achieving inter-rater reliability for facial action and affect intensity ratings is painstaking and labor-intensive. We use computer-vision and machine learning (CVML) to identify patterns of facial actions in 4,648 video recordings of 125 human participants, which show strong correspondences to positive and negative affect intensity ratings obtained from highly trained coders. Our results show that CVML can both (1) determine the importance of different facial actions that human coders use to derive positive and negative affective ratings, and (2) efficiently automate positive and negative affect intensity coding on large facial expression databases. Further, we show that CVML can be applied to individual human judges to infer which facial actions they use to generate perceptual emotion ratings from facial expressions.

show abstract

“…Some users' faces captured during the main experiment are shown in Figure 2, and it is possible to see some different reactions along the simulated flights. The efficiency of the Face Reader software is shown in several researches and publications, being used as a reference regarding to emotion detection from facial expressions on several contexts and applications [29][30][31].…”

Section: Facial Emotion Sensingmentioning

confidence: 99%

Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments

Roza

Postolache

2019

Sensors

View full text Add to dashboard Cite

The present work tries to fill part of the gap regarding the pilots' emotions and their bio-reactions during some flight procedures such as, takeoff, climbing, cruising, descent, initial approach, final approach and landing. A sensing architecture and a set of experiments were developed, associating it to several simulated flights (N f lights = 13) using the Microsoft Flight Simulator Steam Edition (FSX-SE). The approach was carried out with eight beginner users on the flight simulator (N pilots = 8). It is shown that it is possible to recognize emotions from different pilots in flight, combining their present and previous emotions. The cardiac system based on Heart Rate (HR), Galvanic Skin Response (GSR) and Electroencephalography (EEG), were used to extract emotions, as well as the intensities of emotions detected from the pilot face. We also considered five main emotions: happy, sad, angry, surprise and scared. The emotion recognition is based on Artificial Neural Networks and Deep Learning techniques. The Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) were the main methods used to measure the quality of the regression output models. The tests of the produced output models showed that the lowest recognition errors were reached when all data were considered or when the GSR datasets were omitted from the model training. It also showed that the emotion surprised was the easiest to recognize, having a mean RMSE of 0.13 and mean MAE of 0.01; while the emotion sad was the hardest to recognize, having a mean RMSE of 0.82 and mean MAE of 0.08. When we considered only the higher emotion intensities by time, the most matches accuracies were between 55% and 100%.

show abstract

Facial expression analysis with AFFDEX and FACET: A validation study

Cited by 196 publications

References 67 publications

Overview of the Advancements in Automatic Emotion Recognition: Comparative Performance of Commercial Algorithms

Overview of the Advancements in Automatic Emotion Recognition: Comparative Performance of Commercial Algorithms

Using Computer-vision and Machine Learning to Automate Facial Coding of Positive and Negative Affect Intensity

Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments

Contact Info

Product

Resources

About