Engagement Detection through Facial Emotional Recognition Using a Shallow Residual Convolutional Neural Networks

Abstract

Online teaching and learning has recently turned out to be the order of the day, where majority of the learners undergo courses and trainings over the new environment. Learning through these platforms have created a requirement to understand if the learner is interested or not. Detecting engagement of the learners have sought increased attention to create learner centric models that can enhance the teaching and learning experience. The learner will over a period of time in the platform, tend to expose various emotions like engaged, bored, frustrated, confused, angry and other cues that can be classified as engaged or disengaged. This paper proposes in creating a Convolutional Neural Network (CNN) and enabling it with residual connections that can enhance the learning rate of the network and improve the classification on three Indian datasets that predominantly work on classroom engagement models. The proposed network performs well due to introduction of Residual learning that carries additional learning from the previous batch of layers into the next batch, Optimized Hyper Parametric (OHP) setting, increased dimensions of images for higher data abstraction and reduction of vanishing gradient problems resulting in managing overfitting issues. The Residual network introduced, consists of a shallow depth of 50 layers which has significantly produced an accuracy of 91.3% on ISED & iSAFE data while it achieves a 93.4% accuracy on the Daisee dataset. The average accuracy achieved by the classification network is 0.825 according to Cohens Kappa measure.

Detecting Academic Affective States of Learners in Online Learning Environments Using Deep Transfer Learning

“…Based on the literature review, it has been discovered that the majority of researchers focusing on academic affective states utilized the DAiSEE dataset to train their models in order to determine student engagement in learning environments [5,6,8,9,12,36,37]. A commonly noted challenge among researchers working with the DAiSEE dataset is the issue of dataset imbalance [10,14,30,37].…”

“…In the class time, students show positive expressions or emotions such as attention, engagement, and understanding to indicate that he or she is comprehending the material. They express Negative emotions such as confusion, frustration, or boredom to indicate that the student is struggling and may need additional support or clarification [6,7]. Table 1.1 gives the cases when students deliver either positive or negative emotions.…”

Engagement Detection with Multi-Task Training in E-Learning Environments

“…In Reference [28], Whitehill et al showed that automated engagement detectors perform with comparable accuracy to humans. In [3] [19,21,23], authors used models which are based on Convolutional Neural Networks (CNN) and Residual Networks (ResNet) [12]. All the works above considered the engagement detection problem as a multi-class classification problem.…”

Gradient-Based Compact Binary Coding for Facial Expression Recognition

“…Recently, some more methods are developed with the aim of robustness in FER by applying a deep learning-based classifier. M. M. Thiruthuvanathan, and B. Krishnan [41,42] adopted Convolutional Neural Networks (CNNs) and enabled them with residual components to enhance the learning rate of the network. They employed this model to classify classroom engagement through facial expression recognition.…”