Kids’ Emotion Recognition Using Various Deep-Learning Models with Explainable AI

Abstract

Human ideas and sentiments are mirrored in facial expressions. They give the spectator a plethora of social cues, such as the viewer’s focus of attention, intention, motivation, and mood, which can help develop better interactive solutions in online platforms. This could be helpful for children while teaching them, which could help in cultivating a better interactive connect between teachers and students, since there is an increasing trend toward the online education platform due to the COVID-19 pandemic. To solve this, the authors proposed kids’ emotion recognition based on visual cues in this research with a justified reasoning model of explainable AI. The authors used two datasets to work on this problem; the first is the LIRIS Children Spontaneous Facial Expression Video Database, and the second is an author-created novel dataset of emotions displayed by children aged 7 to 10. The authors identified that the LIRIS dataset has achieved only 75% accuracy, and no study has worked further on this dataset in which the authors have achieved the highest accuracy of 89.31% and, in the authors’ dataset, an accuracy of 90.98%. The authors also realized that the face construction of children and adults is different, and the way children show emotions is very different and does not always follow the same way of facial expression for a specific emotion as compared with adults. Hence, the authors used 3D 468 landmark points and created two separate versions of the dataset from the original selected datasets, which are LIRIS-Mesh and Authors-Mesh. In total, all four types of datasets were used, namely LIRIS, the authors’ dataset, LIRIS-Mesh, and Authors-Mesh, and a comparative analysis was performed by using seven different CNN models. The authors not only compared all dataset types used on different CNN models but also explained for every type of CNN used on every specific dataset type how test images are perceived by the deep-learning models by using explainable artificial intelligence (XAI), which helps in localizing features contributing to particular emotions. The authors used three methods of XAI, namely Grad-CAM, Grad-CAM++, and SoftGrad, which help users further establish the appropriate reason for emotion detection by knowing the contribution of its features in it.

New Trends in Emotion Recognition Using Image Analysis by Neural Networks, a Systematic Review

“…There are also situations in which the proposed methods exceed the state-of-the-art performances [ 38 , 39 , 54 , 62 , 74 , 81 , 101 ]. Similarly, context-aware solutions for emotion recognition [ 47 , 49 , 50 , 98 ] or practical solutions [ 37 , 124 , 127 , 130 , 133 , 159 ] demonstrate promising results.…”

“…In addition, predictions are made on three distinct scales, which helps the model improve its accuracy in predicting objects. To achieve feature extraction, in [ 130 ], the authors use the YOLOv3 face detection model.…”

Facial Emotion Recognition-based Engagement De-tection in Autism Spectrum Disorder

“…Rathod et al in [26] proposed a kids' facial emotion recognition system based on using deep-learning models Furthermore, the integration of multimodal signals has shown to improve the accuracy of engagement detection. However, further research is needed to develop more robust and accurate engagement detection-based facial recognition systems that can be applied in real-world scenarios.…”

A Neural Network Architecture for Children’s Audio-Visual Emotion Recognition

“…Despite the difficulties in obtaining emotions data, there are corpora of children's emotional speech in different languages [24][25][26][27] and emotional facial expressions [28] of children. Research is being conducted on the automatic recognition of emotions from children's speech [29,30] and their facial expressions [31]. The accuracy of emotion recognition can be higher when using several modalities [32], for example audio and video, which requires the collection of appropriate audio-visual corpora.…”