Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models

Gupta, Swadha; Kumar, Parteek; Tekchandani, Rajkumar

doi:10.1007/s11042-022-13558-9

Cited by 66 publications

(30 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this phase, the face is detected using the facial detection technique, i.e. Faster R-CNN [27,60]. at each location of the output feature maps of the input image.…”

Section: Face Detection Using Faster R-cnnmentioning

confidence: 99%

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Gupta

Kumar

Tekchandani

2023

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

Due to the COVID-19 crisis, the education sector has been shifted to a virtual environment. Monitoring the engagement level and providing regular feedback during e-classes is one of the major concerns, as this facility lacks in the e-learning environment due to no physical observation of the teacher. According to present study, an engagement detection system to ensure that the students get immediate feedback during e-Learning. Our proposed engagement system analyses the student's behaviour throughout the e-Learning session. The proposed novel approach evaluates three modalities based on the student's behaviour, such as facial expression, eye blink count, and head movement, from the live video streams to predict student engagement in e-learning. The proposed system is implemented based on deep-learning approaches such as VGG-19 and ResNet-50 for facial emotion recognition and the facial landmark approach for eye-blinking and head movement detection. The results from different modalities (for which the algorithms are proposed) are combined to determine the EI (engagement index). Based on EI value, an engaged or disengaged state is predicted. The present study suggests that the proposed facial cues-based multimodal system accurately determines student engagement in real time. The experimental research achieved an accuracy of 92.58% and showed that the proposed engagement detection approach significantly outperforms the existing approaches.

show abstract

“…In this phase, the face is detected using the facial detection technique, i.e. Faster R-CNN [27,60]. at each location of the output feature maps of the input image.…”

Section: Face Detection Using Faster R-cnnmentioning

confidence: 99%

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Gupta

Kumar

Tekchandani

2023

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

show abstract

“…Various technological approaches have applied deep learning algorithms to solve various problems in human life such as in education, robotics, and automotive. In the eld of education, emotion recognition is used to analyze the interests and engagement of learners [11] [73]. Meanwhile, in robotics, it is employed in humanoid robots to foster deeper interactions between robots and humans [13].…”

Section: Discussionmentioning

confidence: 99%

Systematic Literature Review: The Influence and Effectiveness of Deep Learning in Image Processing for Emotion Recognition

Wicaksana,

Davinsi,

Afriyanto

et al. 2024

Preprint

View full text Add to dashboard Cite

In the current digital era, image processing and Emotion Recognition are important topics in the field of artificial intelligence. Deep learning, as one of the most widely used AI techniques in pattern recognition, has shown great potential in addressing these challenges. This research employs a Systematic Literature Review method to collect and analyze previous studies related to deep learning algorithms, namely Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), allowing the researchers to conclude efficient deep learning methods for emotion recognition through image processing. This paper has the result that most studies used CNN to identify emotion from facial expressions, while some studies used RNN. Furthermore, some researchers used combined CNN and RNN to identify emotion from images. Based on the analysis of this research, it is recommended that further studies to take a more holistic approach by considering a wider range of indicators that can be used as signs or signals to analyze a person's emotions. This approach allows for a comprehensive understanding of emotions from multiple perspectives.

show abstract

“…A variety of automatic engagement detection methods are presented by numerous research articles in e-learning environments that incorporate computer vision techniques [2,33]. Due to the direct relationship between facial emotions and perceived attention, SE can be determined from students' facial gestures [35].…”

Section: Literature Reviewmentioning

confidence: 99%

“…SE refers to the focused interaction between educational stakeholders (precisely, instructors and students) with a goal of knowledge delivery [1]. Academic outcomes including grades may be improved if students remain engaged during the learning phase [2]. SE is also considered as an indicator to check whether active learning sessions are taking place during lecture deliverance.…”

Section: Introductionmentioning

confidence: 99%

“…The observational checklist measures SE while relying on the questionnaires filled in by external instructors or observers. Considering some parameters, the observers rate the students' attention in live sessions or examine pre-recorded lecture videos of students but this is not a useful tool for a larger number of students [2]. Automated measurement is the recent focus of researchers that have been studying the real-time computation of SE, starting from an automated engagement tracing method that is presented for estimating SE while considering the timing and accuracy of students' responses [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recognition of Student Engagement State in a Classroom Environment Using Deep and Efficient Transfer Learning Algorithm

et al. 2023

View full text Add to dashboard Cite

A student’s engagement in a real classroom environment usually varies with respect to time. Moreover, both genders may also engage differently during lecture procession. Previous research measures students’ engagement either from the assessment outcome or by observing their gestures in online or real but controlled classroom environments with limited students. However, most works either manually assess the engagement level in online class environments or use limited features for automatic computation. Moreover, the demographic impact on students’ engagement in the real classroom environment is limited and needs further exploration. This work is intended to compute student engagement in a real but least controlled classroom environment with 45 students. More precisely, the main contributions of this work are twofold. First, we proposed an efficient transfer-learning-based VGG16 model with extended layer, and fine-tuned hyperparameters to compute the students’ engagement level in a real classroom environment. Overall, 90% accuracy and 0.5 N seconds computational time were achieved in terms of computation for engaged and non-engaged students. Subsequently, we incorporated inferential statistics to measure the impact of time while performing 14 experiments. We performed six experiments for gender impact on students’ engagement. Overall, inferential analysis reveals the positive impact of time and gender on students’ engagement levels in a real classroom environment. The comparisons were also performed by various transfer learning algorithms. The proposed work may help to improve the quality of educational content delivery and decision making for educational institutions.

show abstract

Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models

Cited by 66 publications

References 57 publications

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Systematic Literature Review: The Influence and Effectiveness of Deep Learning in Image Processing for Emotion Recognition

Recognition of Student Engagement State in a Classroom Environment Using Deep and Efficient Transfer Learning Algorithm

Contact Info

Product

Resources

About