Understanding How CNNs Recognize Facial Expressions: A Case Study with LIME and CEM

Abstract

Recognizing facial expressions has been a persistent goal in the scientific community. Since the rise of artificial intelligence, convolutional neural networks (CNN) have become popular to recognize facial expressions, as images can be directly used as input. Current CNN models can achieve high recognition rates, but they give no clue about their reasoning process. Explainable artificial intelligence (XAI) has been developed as a means to help to interpret the results obtained by machine learning models. When dealing with images, one of the most-used XAI techniques is LIME. LIME highlights the areas of the image that contribute to a classification. As an alternative to LIME, the CEM method appeared, providing explanations in a way that is natural for human classification: besides highlighting what is sufficient to justify a classification, it also identifies what should be absent to maintain it and to distinguish it from another classification. This study presents the results of comparing LIME and CEM applied over complex images such as facial expression images. While CEM could be used to explain the results on images described with a reduced number of features, LIME would be the method of choice when dealing with images described with a huge number of features.

UIBVFEDPlus-Light: Virtual facial expression dataset with lighting

“…To test the possibilities of the new UIBVFEDPlus-Light dataset, the same simple CNN model was trained and tested with the facial expression images generated with the lighting configuration from above without contrast, obtaining an overall accuracy of 0.93 (more detailed information about the training and testing followed procedure can be found in Castillo Torres et al [21]). The CNN model trained with the images with the lighting configuration from above without contrast has also been tested with all the remaining lighting configurations.…”

Behavior Analysis of Photo-Taking Tourists at Attraction-Level Using Deep Learning and Latent Dirichlet Allocation in Conjunction With Kernel Density Estimation

“…Likewise, computer image processing technology is also evolving and becoming increasingly sophisticated [44]. Consequently, various fields are adopting Artificial Intelligence (AI) for image recognition purposes [45]. The application of AI-based big data processing techniques in the analysis and recognition of location photographs captured by tourists overcomes the constraints of manual approaches and offers technological assistance in extracting intricate visual and semantic data from such images [46].…”

Research on Spatio-Temporal Characteristics of Tourists’ Landscape Perception and Emotional Experience by Using Photo Data Mining

“…In the past, due to the limitations of visual semantic mining technology, most of the research on the visual content of photos was based on manual recognition and classification coding methods, which were inefficient in processing data volume, and the research results were also extremely subjective [10]. Along with the rapid development of computer deep learning and big data mining technology, computer image processing technology is becoming more and more mature, and artificial intelligence is widely used in various fields of image recognition [11]. The use of artificial intelligence big data processing methods to identify and parse the contents of tourists' location photos breaks through the limitations of manual methods and provides technical support for mining the complex visual semantic information of these images [8].…”