Real-Time Facial Expression Recognition using Facial Landmarks and Neural Networks

Haghpanah, Mohammad Amin; Saeedizade, Ehsan; Masouleh, Mehdi Tale; Kalhor, Ahmad

doi:10.1109/mvip53647.2022.9738754

Cited by 8 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[16] 99.16 83.67 Ref. [36] 97.83 -VGG19 + SVM [37] 98.76 -HistNet [38] 98.47 83.41 FMN [55] 98.61 81.39 DLP-CNN [56] 95.78 78.46 IPA2LT [57] 92.45 65.61 Ref. [58] 96.12 -PAU-Net (type I) [59] -85.42 PAU-Net (type II) [59] -85.89 DSN-DF [60] 98.9 79.33 VGG19+ our network 99.2031 98.1378…”

Section: Methods Ck+ MMImentioning

confidence: 99%

“…Due to the limitations of traditional methods, the extracted feature information was not comprehensive enough and it was necessary to add auxiliary measures to improve the recognition accuracy. In [15,[36][37][38]55,[58][59][60], deep learning methods were used to conduct global learning, local learning, or a combination of global and local learning. However, due to the small number of samples and the unbalanced sample distributions in the MMI and CK+ datasets, redundant learning and repetitive learning occurred from time to time.…”

Section: Methods Ck+ MMImentioning

confidence: 99%

See 1 more Smart Citation

The Facial Expression Data Enhancement Method Induced by Improved StarGAN V2

Han

2023

Symmetry

View full text Add to dashboard Cite

Due to the small data and unbalanced sample distribution in the existing facial emotion datasets, the effect of facial expression recognition is not ideal. Traditional data augmentation methods include image angle modification, image shearing, and image scrambling. The above approaches cannot solve the problem that is the high similarity of the generated images. StarGAN V2 can generate different styles of images across multiple domains. Nevertheless, there are some defects in gener-ating these facial expression images, such as crooked mouths and fuzzy facial expression images. To service such problems, we improved StarGAN V2 by solving the drawbacks of creating pictures that apply an SENet to the generator of StarGAN V2. The generator’s SENet can concentrate at-tention on the important regions of the facial expression images. Thus, this makes the generated symmetrical expression image more obvious and easier to distinguish. Meanwhile, to further im-prove the quality of the generated pictures, we customized the hinge loss function to reconstruct the loss functions that increase the boundary of real and fake images. The created facial expression pictures testified that our improved model could solve the defects in the images created by the original StarGAN V2. The experiments were conducted on the CK+ and MMI datasets. The correct recognition rate of the facial expressions on the CK+ was 99.2031%, which is a 1.4186% higher accuracy than that of StarGAN V2. The correct recognition rate of the facial expressions on the MMI displays was 98.1378%, which is 5.059% higher than that of the StarGAN V2 method. Furthermore, contrast test outcomes proved that the improved StarGAN V2 performed better than most state-of-the-art methods.

show abstract

Section: Methods Ck+ MMImentioning

confidence: 99%

Section: Methods Ck+ MMImentioning

confidence: 99%

The Facial Expression Data Enhancement Method Induced by Improved StarGAN V2

Han

2023

Symmetry

View full text Add to dashboard Cite

show abstract

“…In order to categorize the presented statement using Euclidean distances, the input feature vectors are trained in a Multi-Layer Perception (MLP) neural network. The KDEF database provided the dataset used for this investigation [17]. Raman demonstrates the operation of a speculative method for instantly detecting a person's mood.…”

Section: Pure Science and Technology Applications (Scug-psta-2022)mentioning

confidence: 99%

The Impact of Emotions on the Speed and Score of Computer Games

Sharif

Ghareb

Muhamad

2022

PJBAS

View full text Add to dashboard Cite

This Research has examined how video game affects the performance of the player. Unity tool has been used to create an interactive game with several ball objects that accrue additional points for each participant. Different methods have been used to strike the balls, using the mouse, keyboard, combination of both, and facial emotion. Image processing techniques have been used to match the facial expression with the EMOGI of the balls the players hit. The performance of the game has been affected by facial expressions. The study demonstrates how widely used, and highly engaging face-emotion techniques are in video games. The usage of visual observation concerning traditional input control and emotion input has been examined, and the differences are demonstrated. Our results show that facial emotions improve the technology for video games; in our game, we tested three types of facial emotions: sad, happy, and angry.

show abstract

“…In [20], the author proposed to learn object structural representations by using landmarks as a complementary feature to the pretrained deep representation in recognizing visual attributes. In [21], the authors proposed to extract facial geometric features by using landmarks, and both geometric and texture-based features can be used to improve the accuracy of facial expression recognition. Such a method can also help the model generalize to new faces, which are out of the training set.…”

Section: Introductionmentioning

confidence: 99%

LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

Xue¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Lipreading refers to understanding and further translating the speech of a speaker in the video into natural language. State-of-the-art lipreading methods excel in interpreting overlap speakers, i.e., speakers appear in both training and inference sets. However, generalizing these methods to unseen speakers incurs catastrophic performance degradation due to the limited number of speakers in training bank and the evident visual variations caused by the shape/color of lips for different speakers. Therefore, merely depending on the visible changes of lips tends to cause model overfitting. To address this problem, we propose to use multi-modal features across visual and landmarks, which can describe the lip motion irrespective to the speaker identities. Then, we develop a sentence-level lipreading framework based on visual-landmark transformers, namely LipFormer. Specifically, LipFormer consists of a lip motion stream, a facial landmark stream, and a cross-modal fusion. The embeddings from the two streams are produced by self-attention, which are fed to the cross-attention module to achieve the alignment between visuals and landmarks. Finally, the resulting fused features can be decoded to output texts by a cascade seq2seq model. Experiments demonstrate that our method can effectively enhance the model generalization to unseen speakers.

show abstract

Real-Time Facial Expression Recognition using Facial Landmarks and Neural Networks

Cited by 8 publications

References 16 publications

The Facial Expression Data Enhancement Method Induced by Improved StarGAN V2

The Facial Expression Data Enhancement Method Induced by Improved StarGAN V2

The Impact of Emotions on the Speed and Score of Computer Games

LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

Contact Info

Product

Resources

About