Which CNNs and Training Settings to Choose for Action Unit Detection? A Study Based on a Large-Scale Dataset

Bishay, Mina; Ghoneim, Ahmed; Ashraf, Mohamed; Mavadati, Mohammad

doi:10.1109/fg52635.2021.9667083

Cited by 4 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This proposal involves incorporating the values obtained from the abovementioned analyses into a convolutional neural network, resulting in improved precision values, particularly for the emotions of sadness, surprise, and anger. Despite achieving better results through transfer learning, these emotions still exhibited a true positive rate (TPR) of less than 67% [17].…”

Section: Set Valuesmentioning

confidence: 99%

See 1 more Smart Citation

Emotion Recognition in Individuals with Down Syndrome: A Convolutional Neural Network-Based Algorithm Proposal

2023

View full text Add to dashboard Cite

This research introduces an algorithm that automatically detects five primary emotions in individuals with Down syndrome: happiness, anger, sadness, surprise, and neutrality. The study was conducted in a specialized institution dedicated to caring for individuals with Down syndrome, which allowed for collecting samples in uncontrolled environments and capturing spontaneous emotions. Collecting samples through facial images strictly followed a protocol approved by certified Ethics Committees in Ecuador and Colombia. The proposed system consists of three convolutional neural networks (CNNs). The first network analyzes facial microexpressions by assessing the intensity of action units associated with each emotion. The second network utilizes transfer learning based on the mini-Xception architecture, using the Dataset-DS, comprising images collected from individuals with Down syndrome as the validation dataset. Finally, these two networks are combined in a CNN network to enhance accuracy. The final CNN processes the information, resulting in an accuracy of 85.30% in emotion recognition. In addition, the algorithm was optimized by tuning specific hyperparameters of the network, leading to a 91.48% accuracy in emotion recognition accuracy, specifically for people with Down syndrome.

show abstract

Section: Set Valuesmentioning

confidence: 99%

“…On the topic of facial analysis, microexpressions have received significant attention. Bishay [17] conducted a study investigating the utilization of various convolutional neural networks to detect action units (AUs). The study compared ten CNNs, including ResNet, VGG, and DenseNet.…”

Section: Introductionmentioning

confidence: 99%

Emotion Recognition in Individuals with Down Syndrome: A Convolutional Neural Network-Based Algorithm Proposal

2023

View full text Add to dashboard Cite

show abstract

“…RAF-DB is a real-world dataset that includes images of faces collected from the Internet and manually labelled with crowdsourcing. The neural network architecture we trained is similar to the baseline model used in the study of Bishay et al [45], a shallow convolutional neural network (CNN) including four convolutional layers with 32, 32, 64, and 64 filters respectively and the ReLU activation function. The first three layers were followed by a max-pooling layer with a 2×2 filter, and the last one by a flatten layer.…”

Section: B Analysis Of the Collected Emotional Facial Image Datamentioning

confidence: 99%

Interpretable Explainability in Facial Emotion Recognition and Gamification for Data Collection

Shingjergji

İren

Böttger

et al. 2022

2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

Training facial emotion recognition models requires large sets of data and costly annotation processes. To alleviate this problem, we developed a gamified method of acquiring annotated facial emotion data without an explicit labeling effort by humans. The game, which we named Facegame, challenges the players to imitate a displayed image of a face that portrays a particular basic emotion. Every round played by the player creates new data that consists of a set of facial features and landmarks, already annotated with the emotion label of the target facial expression. Such an approach effectively creates a robust, sustainable, and continuous machine learning training process. We evaluated Facegame with an experiment that revealed several contributions to the field of affective computing. First, the gamified data collection approach allowed us to access a rich variation of facial expressions of each basic emotion due to the natural variations in the players' facial expressions and their expressive abilities. We report improved accuracy when the collected data were used to enrich well-known in-the-wild facial emotion datasets and consecutively used for training facial emotion recognition models. Second, the natural language prescription method used by the Facegame constitutes a novel approach for interpretable explainability that can be applied to any facial emotion recognition model. Finally, we observed significant improvements in the facial emotion perception and expression skills of the players through repeated game play.

show abstract

“…In this section we describe the dataset, CNN architecture, and experimental settings used for building and training our AU detector. Most of the settings in our AU detection architecture are chosen based on the recommendations given in [17,18]. AU Dataset.…”

Section: Au Detectionmentioning

confidence: 99%

Automatic Detection of Sentimentality from Facial Expressions

Bishay

Turcot

Page

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Emotion recognition has received considerable attention from the Computer Vision community in the last 20 years. However, most of the research focused on analyzing the six basic emotions (e.g. joy, anger, surprise), with a limited work directed to other affective states. In this paper, we tackle sentimentality (strong feeling of heartwarming or nostalgia), a new emotional state that has few works in the literature, and no guideline defining its facial markers. To this end, we first collect a dataset of 4.9K videos of participants watching some sentimental and non-sentimental ads, and then we label the moments evoking sentimentality in the ads. Second, we use the ad-level labels and the facial Action Units (AUs) activation across different frames for defining some weak framelevel sentimentality labels. Third, we train a Multilayer Perceptron (MLP) using the AUs activation for sentimentality detection. Finally, we define two new ad-level metrics for evaluating our model performance. Quantitative and qualitative results show promising results for sentimentality detection. To the best of our knowledge this is the first work to address the problem of sentimentality detection.

show abstract

Which CNNs and Training Settings to Choose for Action Unit Detection? A Study Based on a Large-Scale Dataset

Cited by 4 publications

References 27 publications

Emotion Recognition in Individuals with Down Syndrome: A Convolutional Neural Network-Based Algorithm Proposal

Emotion Recognition in Individuals with Down Syndrome: A Convolutional Neural Network-Based Algorithm Proposal

Interpretable Explainability in Facial Emotion Recognition and Gamification for Data Collection

Automatic Detection of Sentimentality from Facial Expressions

Contact Info

Product

Resources

About