Student behavior analysis to measure engagement levels in online learning environments

Abstract

After the COVID-19 pandemic, no one refutes the importance of smart online learning systems in the educational process. Measuring student engagement is a crucial step towards smart online learning systems. A smart online learning system can automatically adapt to learners' emotions and provide feedback about their motivations. In the last few decades, online learning environments have generated tremendous interest among researchers in computer-based education. The challenge that researchers face is how to measure student engagement based on their emotions. There has been an increasing interest towards computer vision and camera-based solutions as technology that overcomes the limits of both human observations and expensive equipment used to measure student engagement. Several solutions have been proposed to measure student engagement, but few are behavior-based approaches. In response to these issues, in this paper, we propose a new automatic multimodal approach to measure student engagement levels in real time. Thus, to offer robust and accurate student engagement measures, we combine and analyze three modalities representing students' behaviors: emotions from facial expressions, keyboard keystrokes, and mouse movements. Such a solution operates in real time while providing the exact level of engagement and using the least expensive equipment possible. We validate the proposed multimodal approach through three main experiments, namely single, dual, and multimodal research modalities in novel engagement datasets. In fact, we build new and realistic student engagement datasets to validate our contributions. We record the highest accuracy value (95.23%) for the multimodal approach and the lowest value of "0.04" for mean square error (MSE).

Automatic modeling of student characteristics with interaction and physiological data using machine learning: A review

“…The research achieved the best performance in terms of accuracy among all the reviewed papers that predicted student engagement levels using physiological data. To provide robust and more effective student engagement measures, Altuwairqi et al ( 2021 ) combined learning interaction and physiological data (facial expression data, mouse movement and keystrokes) of students to build a naïve bayes model that predicted student engagement with an accuracy of 95.2%. The accuracy level of the two approaches suggested that automatic student engagement detection can effectively track issues on engagement in learning environments.…”

“…Based on this literature review, the most predominant approach for detecting engagement using physiological data is facial expression tracking. A number of studies (Monkaresi et al, 2017 ; Dubbaka and Gopalan, 2020 ; Mohamad Nezami et al, 2020 ; Altuwairqi et al, 2021 ) analyzed student engagement during interaction with online educational systems using facial expressions. The studies focused on capturing and analyzing facial expressions to identify disengagement and engagement states of students.…”

Predicting Online Student Effort with Accelerometer, Heart Rate Sensors, and Camera Using Random Forest Regression Model

“…The methods in [28,29,30,31] predict an online/offline student 's engagement on an ordinary scale with 3 or more levels. In [28], the engagement is predicted by 3 levels through a naive-Bayes classification model whose inputs are mouse gesture, keystroke, facial expression feature extracted by webcam image. The prediction from PC utilizing behavior and webcam image has the advantage of noninvasiveness and simplicity of implementation.…”

Wearables for Engagement Detection in Learning Environments: A Review

“…Other works centered around student emotional state detection analyze and process signals from Electroencephalogram (EEG), Electromyogram (EMG), Electrocardiography (ECG), Electrodermal activity (EDA), heart rate variability, skin temperature, blood volume pulse, respiration, or Electrodermography (EDG)/galvanic skin response (GSR) [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ]. Researchers [ 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 ] report the use of deep learning and machine learning (ML) techniques for emotion classification. Finally, other techniques rely on emotion recognition via computer vision [ 22 , 41 , 48 , 49 , 50 ], linguistic semantic approaches [ 51 ], and biological features [ 52 ].…”