Margin-Mix: Semi-Supervised Learning for Face Expression Recognition

Florea, Corneliu; Badea, Mihai-Sorin; Florea, Laura; Racoviteanu, Andrei; Vertan, Constantin

doi:10.1007/978-3-030-58592-1_1

Cited by 14 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although existing studies have evaluated models exclusively against the entirety of the CAFE data set [34][35][36][37][38][39], we additionally evaluated them on Subset A and Subset B of CAFE, as defined by the authors of the data set. Subset A contains images that were identified with an accuracy of 60% or above by 100 adult participants [54], with a Cronbach α internal consistency score of .82 (versus .77 for the full CAFE data set).…”

Section: Model Evaluationmentioning

confidence: 99%

“…The increasing use of signals from sensors on mobile devices, such as the selfie camera, opens many possibilities for real-time analysis of image data for continuous phenotyping and repeated diagnoses in home settings [18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33]. However, facial emotion classifiers and the underlying data sets on which they are trained have been tailored to neurotypical adults, as demonstrated by repeatedly low performance on image data sets of pediatric emotion expressions [34][35][36][37][38][39].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study (Preprint)

Washington¹,

Kalantarian²,

Kent³

et al. 2020

Preprint

View full text Add to dashboard Cite

BACKGROUND Automated emotion classification could aid those who struggle to recognize emotion, including children with developmental behavioral conditions such as autism. However, most computer vision emotion recognition models are trained on adult affect and therefore underperform when used on child faces. OBJECTIVE We designed a strategy to gamify the collection and the labeling of child affect data in an effort to boost the performance of automatic child emotion detection to a level closer to what will be needed for digital healthcare approaches. METHODS We leveraged our prototype therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Independently, we created a secure web interface to gamify the human labeling effort HollywoodSquares, tailored for use by any qualified labeler. We gathered and labeled 2,155 videos, 39,968 emotion frames, and 106,001 labels on all images. With this drastically expanded pediatric emotion centric database (>30x larger than existing public pediatric affect datasets), we trained a pediatric emotion classification convolutional neural network (CNN) classifier of happy, sad, surprised, fearful, angry, disgust, and neutral expressions in children. RESULTS The classifier achieved 66.9% balanced accuracy and 67.4% F1-score on the entirety of CAFE as well as 79.1% balanced accuracy and 78.0% F1-score on CAFE Subset A, a subset containing at least 60% human agreement on emotions labels. This performance is at least 10% higher than all previously published classifiers, the best of which reached 56.0% balanced accuracy even when combining “anger” and “disgust” into a single class. CONCLUSIONS This work validates that mobile games designed for pediatric therapies can generate high volumes of domain-relevant datasets to train state of the art classifiers to perform tasks highly relevant to precision health efforts.

show abstract

Section: Model Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study (Preprint)

Washington¹,

Kalantarian²,

Kent³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…They explore different methods, such as analysing Facebook status updates to detect depression ( Ali et al, 2022 ) and predicting suicide risk and mental health problems on Twitter through multitask learning ( Benton et al, 2017 ) and depression prediction ( de Souza et al, 2022 ) on Reddit. Regarding facial expression recognition, some works ( Florea et al, 2019 ; Giannakakis et al, 2017 ; Huang et al, 2016 ) also address anxiety detection using machine learning techniques. However, all of these works use either pre-processed text datasets, which provide textual features of the mental disorder, or facial prediction datasets, which contain images of individuals with anxiety, for training.…”

Section: Discussionmentioning

confidence: 99%

“…Recently, a comprehensive review of the most commonly used classical and neural network strategies for interpreting and recognising facial expressions of emotion ( Canal et al, 2022 ) has been conducted. In terms of machine learning techniques for facial emotion recognition, Florea et al (2019) propose to predict anxiety/stress in children using the Annealed Label Transfer method. In addition, Giannakakis et al (2017) develop a framework for the detection and analysis of stress/anxiety, focusing on non-voluntary and semi-voluntary facial signals.…”

Section: Introductionmentioning

confidence: 99%

Internet-based identification of anxiety in university students using text and facial emotion analysis

Guerrero,

Avila,

da Silva

et al. 2023

Internet Interventions

View full text Add to dashboard Cite

“…SSL explores both labeled data and unlabeled data simultaneously in order to mitigate the requirement for labeled data. Many SSL models have presented excellent performance in computer vision and machine learning [28], [29], and they can be categorized into two broad categories: pseudo-labeling [30] and consistency regularization [31], [32]. For a systematic review of SSL in FER, please refer to [33].…”

Section: B Deep Learning-based Semi-supervised Learningmentioning

confidence: 99%

Rethinking Pseudo-Labeling for Semi-Supervised Facial Expression Recognition With Contrastive Self-Supervised Learning

Fang

Han

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Self-supervised learning for semi-supervised facial expression recognition aims to avoid the need to collect expensive labeled facial expression data. Existing methods demonstrate an impressive performance boost, but they artificially assume that small labeled facial expression data and large unlabeled facial expression data are from the same class distribution. In a more realistic scenario, when utilizing facial expression data from a large face recognition database as unlabeled data, there will be a mismatch distribution between the two sets of data. This often results in severe performance degradation due to incorrect propagation of unlabeled data from unrelated sources. In this paper, we address the class distribution mismatch problem in deep semi-supervised learning-based facial expression recognition. Specifically, we propose a silhouette coefficient-based contrast clustering algorithm, which determines the degree of separation between clusters by examining the intra-cluster and inter-cluster distances to accurately detect out-of-distribution data. In addition, we propose a pseudo-labeling rethinking strategy that matches the soft pseudo-labels estimated from a fine-tuned network to the contrast clustering to produce reliable pseudolabels. Experiments on three in-the-wild datasets, RAF-DB, FERPlus and AffectNet, demonstrate the effectiveness of our method, and our approach performs well compared to state-of-the-art methods.INDEX TERMS Facial expression recognition, semi-supervised learning, contrastive self-supervised learning, out-of-distribution data.

show abstract

Margin-Mix: Semi-Supervised Learning for Face Expression Recognition

Cited by 14 publications

References 26 publications

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study (Preprint)

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study (Preprint)

Internet-based identification of anxiety in university students using text and facial emotion analysis

Rethinking Pseudo-Labeling for Semi-Supervised Facial Expression Recognition With Contrastive Self-Supervised Learning

Contact Info

Product

Resources

About