A Lightweight Facial Emotion Recognition System Using Partial Transfer Learning for Visually Impaired People

Abstract

The inability to perceive visual and other non-verbal cues for individuals with visual impairment can pose a significant challenge for their correct conversational interactions and can be an impediment for various daily life activities. Recent advancements in computational resources, particularly the computer vision capabilities can be utilized to design effective applications for visually impaired people (VIP).Among various assistive technologies, automated facial impression recognition with realtime accurate interpretation can be proven useful to tackle the above problem. Using such approach, facial emotions (e.g., sad, happy) can be robustly recognized and conveyed to the associated individuals. In this paper, a partial transfer learning approach was adopted utilizing a custom trained Convolutional Neural Network (CNN) for facial emotion recognition. A novel model that transfers features from one dataset to another is proposed. This model enables the transfer of features learned from a small number of instances to solve new challenges instances. Using the proposed approach based on a newly trained CNN, a portable lightweight facial expression recognition system with wireless connectivity and high detection accuracy was constructed and targeted specifically for VIP. The proposed recognition model provides a notable improvement over the current state-of-the-art, by providing the highest recognition accuracy of 82.1% on the enhanced Facial Expression Recognition 2013 (FER2013) FER2013 dataset. Moreover, with only 1.49M parameters, the model is operable on edge devices with limited memory and processing power. Overall, three labeled emotions happy, sad, surprise were recognized by the model with high accuracy whereas a relatively lower accuracy rate for anger, disgust, fear was noticed with higher misclassification labels for sad.

Facial Expression Recognition Based on Deep Learning: A Review

“…Dina et al [23] Addresses the challenge faced by visually impaired individuals in perceiving non-verbal cues, hindering their conversational interactions and daily activities. Leveraging advancements in computer vision, the authors propose a partial transfer learning approach using a custom-trained Convolutional Neural Network (CNN) for automated facial emotion recognition.…”

“…The proposed partial transfer learning approach addresses the challenges faced by visually impaired individuals, providing a notable recognition accuracy of 82.1% on the FER2013 dataset. The focus on custom-trained CNNs for automated facial emotion recognition contributes to a portable, lightweight, and accurate assistive solution [23].…”

FBANet: Transfer Learning for Depression Recognition Using a Feature-Enhanced Bi-Level Attention Network

“…For example, Oquab et al [ 54 ] used transfer learning to first train a traditional CNN model on the ImageNet dataset [ 55 ] and then performed classification on the Pascal VOC dataset [ 56 ]. Shehada et al [ 57 ] used transfer learning for facial expression recognition. They first pre-trained the model on the FER2013 dataset [ 58 ] and then fine-tuned the model on the CK+ dataset [ 59 ] to improve the model’s performance.…”