Facial Emotion Recognition in Verbal Communication Based on Deep Learning

Abstract

Facial emotion recognition from facial images is considered a challenging task due to the unpredictable nature of human facial expressions. The current literature on emotion classification has achieved high performance over deep learning (DL)-based models. However, the issue of performance degradation occurs in these models due to the poor selection of layers in the convolutional neural network (CNN) model. To address this issue, we propose an efficient DL technique using a CNN model to classify emotions from facial images. The proposed algorithm is an improved network architecture of its kind developed to process aggregated expressions produced by the Viola–Jones (VJ) face detector. The internal architecture of the proposed model was finalised after performing a set of experiments to determine the optimal model. The results of this work were generated through subjective and objective performance. An analysis of the results presented herein establishes the reliability of each type of emotion, along with its intensity and classification. The proposed model is benchmarked against state-of-the-art techniques and evaluated on the FER-2013, CK+, and KDEF datasets. The utility of these findings lies in their application by law-enforcing bodies in smart cities.

Advancing Facial Expression Recognition in Online Learning Education Using a Homogeneous Ensemble Convolutional Neural Network Approach

“…Average pooling determines the average of the Convolution neural networks (CNNs) have revolutionized the computer vision field. These networks are extremely effective for a variety of image and video recognition tasks due to their ability to automatically learn and extract key features from videos [29] or image [30,31]. CNN has yielded other models in deep learning, indicating the potential of deep convolution neural networks.…”

Emotion Detection Based on Pupil Variation

“…According to Ekman's emotion theory, the six primary human emotions comprise happiness, sadness, fear, disgust, anger, and surprise [25][26][27]. The definition of each emotion is listed as follows:…”

Algorithms used for facial emotion recognition: a systematic review of the literature

“…A review of all the papers was applied where it was determined that the best method, model and dataset is [VGG19+ our network] + CK+ with an accuracy degree of 99.20%. (35) [[TLF-ResNet18] SVM + AFFECTNET 7 Emociones] 66,37% CK+ Alsharekh, M. F., 2023 (22) [Viola-Jones + CK+] 90,98% Assiri, B., & Hossain, M. A., 2023 (32) [CNN + ARs] + Precisión punta de la nariz 94,51% Dudekula, U., & Purnachand, N. 2023 (40) [NVIDIA Jetson Nano + Entorno en tiempo real + OpenCV] + CK+ 95,60% [NVIDIA Jetson Nano + VGG-19] + CK+ 98,40% [NVIDIA Jetson Nano + Xception] + CK+ 97,10% Gupta et al, 2023 (37) [VGG19] + CK+ 90,14% Han, B., & Hu, M., 2023 (45) [VGG19+ nuestra red] + CK+ 99,20% FER2013 Alsharekh, M. F., 2022 (22) [Viola-Jones + FER-2013] 89,20% Gupta et al, 2023 (37) [Inception-V3] + FER-2013 89,11% JAFFEE Haider et al, 2023 (35) [[TLF-ResNet18 SVM + JAFFE] 98,44% KDEF Alsharekh, M. F., 2022 (22) [Viola-Jones + KDEF] 94,04% MLF-W-FER ELsayed et al, 2023 (34) [AFER] 70,76% Shahzad et al, 2023 (42) [AlexNet + MLF-W-FER + PreEntrenada] + FC8 55,64% [VGG-16 + MLF-W-FER + PreEntrenada] + FC8 56,73% MMI Haider et al, 2023 (35) [[TLF-ResNet18] SVM + MMI] 99,02% Han, B., & Hu, M., 2023 (45) [VGG19+ nuestra red] + MMI 98% RAF-DB Gupta et al, 2023 (37) [ResNet-50] + RAF-DB 92,32% SAVEE Singh et al, 2023 (44) [3DCNN + ConvLSTM] + SAVEE 98,83% [3DCNN] + SAVEE 97,92%…”