Deep Siamese Neural Networks for Facial Expression Recognition in the Wild

Hayale, Wassan; Negi, Pooran; Mahoor, Mohammad H.

doi:10.1109/taffc.2021.3077248

Cited by 20 publications

(6 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mukhiddinov et al [ 74 ] explore emotion detection in faces with masks, focusing on the upper part of the face, and introduce a two-branch CNN-based model for low-light image enhancement, achieving 69.3% accuracy on AffectNet. Similarly, Hayale et al [ 86 , 140 ] implement a Siamese model with two branches using DenseNet121 in each, focusing on mapping images in feature space for improved emotion detection, and demonstrating accuracies up to 98.95% on CFEE-7 dataset. This approach highlights the effectiveness of bifurcated networks in handling complex facial recognition tasks in varied conditions.…”

Section: Resultsmentioning

confidence: 99%

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

Pereira,

Mendes,

Ribeiro

et al. 2024

Sensors

View full text Add to dashboard Cite

Emotion recognition has become increasingly important in the field of Deep Learning (DL) and computer vision due to its broad applicability by using human–computer interaction (HCI) in areas such as psychology, healthcare, and entertainment. In this paper, we conduct a systematic review of facial and pose emotion recognition using DL and computer vision, analyzing and evaluating 77 papers from different sources under Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our review covers several topics, including the scope and purpose of the studies, the methods employed, and the used datasets. The scope of this work is to conduct a systematic review of facial and pose emotion recognition using DL methods and computer vision. The studies were categorized based on a proposed taxonomy that describes the type of expressions used for emotion detection, the testing environment, the currently relevant DL methods, and the datasets used. The taxonomy of methods in our review includes Convolutional Neural Network (CNN), Faster Region-based Convolutional Neural Network (R-CNN), Vision Transformer (ViT), and “Other NNs”, which are the most commonly used models in the analyzed studies, indicating their trendiness in the field. Hybrid and augmented models are not explicitly categorized within this taxonomy, but they are still important to the field. This review offers an understanding of state-of-the-art computer vision algorithms and datasets for emotion recognition through facial expressions and body poses, allowing researchers to understand its fundamental components and trends.

show abstract

Section: Resultsmentioning

confidence: 99%

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

Pereira,

Mendes,

Ribeiro

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…Compared with other FSL algorithms (e.g., the Matching Network [29] used by Jiang et al), Siamese network uses the pair-by-pair learning structure (learn the difference between two samples in two categories) instead of using the one-to-many learning structure (learn the difference between one sample and samples in other categories). It has been widely used for emotion recognition because of its simple and interpretable structure [46]. For example, Hayale et al [46] use the Deep Siamese Neural (DSN) network [47] to recognize 6 basic emotions by facial expressions.…”

Section: B Few-shot Learning Based Emotion Recognitionmentioning

confidence: 99%

“…It has been widely used for emotion recognition because of its simple and interpretable structure [46]. For example, Hayale et al [46] use the Deep Siamese Neural (DSN) network [47] to recognize 6 basic emotions by facial expressions. For uni-dimensional signals, DSN is also used by Feng et al [48] to predict low/medium/high arousal using speech signals.…”

Section: B Few-shot Learning Based Emotion Recognitionmentioning

confidence: 99%

Few-Shot Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Zhang

Ali

Hanjalic

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Fine-grained emotion recognition can model the temporal dynamics of emotions, which is more precise than predicting one emotion retrospectively for an activity (e.g., video clip watching). Previous works require large amounts of continuously annotated data to train an accurate recognition model, however experiments to collect such large amounts of continuously annotated physiological signals are costly and time-consuming. To overcome this challenge, we propose an Emotion recognition algorithm based on Deep Siamese Networks (EmoDSN) which can rapidly converge on a small amount of training data, typically less than 10 samples per class (i.e., < 10 shot). EmoDSN recognizes fine-grained valence and arousal (V-A) labels by maximizing the distance metric between signal segments with different V-A labels. We tested EmoDSN on three different datasets collected in three different environments: desktop, mobile and HMD-based virtual reality, respectively. The results from our experiments show that EmoDSN achieves promising results for both one-dimension binary (high/low V-A, 1D-2C) and two-dimensional 5-class (four quadrants of V-A space + neutral, 2D-5C) classification. We get an averaged accuracy of 76.04%, 76.62% and 57.62% for 1D-2C valence, 1D-2C arousal, and 2D-5C, respectively, by using only 5 shots of training data. Our experiments show that EmoDSN can achieve better results if we select training samples from the changing points of emotion or the ending moments of video watching.

show abstract

“…Discriminant distribution-agnostic loss [39] enforces the inter-class dissimilarity which can be useful while dealing with extremely imbalanced datasets. Hayale et al [40] proposed an algorithm for automated FER to preserve the local structure of images in the embedding similarity space.…”

Section: Related Workmentioning

confidence: 99%

Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild

Fard

Mahoor

2022

IEEE Access

View full text Add to dashboard Cite

Automated Facial Expression Recognition (FER) in the wild using deep neural networks is still challenging due to intra-class variations and inter-class similarities in facial images. Deep Metric Learning (DML) is among the widely used methods to deal with these issues by improving the discriminative power of the learned embedded features. This paper proposes an Adaptive Correlation (Ad-Corre) Loss to guide the network towards generating embedded feature vectors with high correlation for within-class samples and less correlation for between-class samples. Ad-Corre consists of 3 components called Feature Discriminator, Mean Discriminator, and Embedding Discriminator. We design the Feature Discriminator component to guide the network to create the embedded feature vectors to be highly correlated if they belong to a similar class, and less correlated if they belong to different classes. In addition, the Mean Discriminator component leads the network to make the mean embedded feature vectors of different classes to be less similar to each other. We use Xception network as the backbone of our model, and contrary to previous work, we propose an embedding feature space that contains k feature vectors. Then, the Embedding Discriminator component penalizes the network to generate the embedded feature vectors, which are dissimilar. We trained our model using the combination of our proposed loss functions called Ad-Corre Loss jointly with the cross-entropy loss. We achieved a very promising recognition accuracy on AffectNet, RAF-DB, and FER-2013. Our extensive experiments and ablation study indicate the power of our method to cope well with challenging FER tasks in the wild.

show abstract

Deep Siamese Neural Networks for Facial Expression Recognition in the Wild

Cited by 20 publications

References 65 publications

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

Few-Shot Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild

Contact Info

Product

Resources

About