Students Engagement Level Detection in Online e-Learning Using Hybrid EfficientNetB7 Together With TCN, LSTM, and Bi-LSTM

Abstract

Students engagement level detection in online e-learning has become a crucial problem due to the rapid advance of digitalization in education. In this paper, a novel Videos Recorded for Egyptian Students Engagement in E-learning (VRESEE) dataset is introduced for students engagement level detection in online e-learning. This dataset is based on an experiment conducted on a group of Egyptian college students by video recording them during online e-learning sessions. Each recorded video is labeled with a value from 0 to 3 representing the level of engagement of each student during the online session. Moreover, three new hybrid end-to-end deep learning models have been proposed for detecting student's engagement level in an online e-learning video. These models are evaluated using the VRESEE dataset and also using a public Dataset for the Affective States in E-Environment (DAiSEE). The first proposed hybrid model uses EfficientNet B7 together with Temporal Convolution Network (TCN) and achieved an accuracy of 64.67% on DAiSEE and 81.14% on VRESEE. The second model uses a hybrid EfficientNet B7 along with Long Short Term Memory (LSTM) and reached an accuracy of 67.48% on DAiSEE and 93.99% on VRESEE. Finally, the third hybrid model uses EfficientNet B7 along with a Bidirectional LSTM and achieved an accuracy of 66.39% on DAiSEE and 94.47% on VRESEE. The results of the first, second and third proposed models outperform the results of currently existing models by 1.08%, 3.89%, and 2.8% respectively in students engagement level detection.

Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

“…The proposed lightweight BoS model, ignoring the order of events, achieved an accuracy of 66.58%, one of the highest accuracies among the existing works. Only the accuracies of two complex methods are above the proposed method, Hybrid EfficientNetB7 with LSTM [35], and affect-driven ordinal TCN [37]. By comparing the proposed method to previous methods, it becomes clear that incorporating the order of states in engagement measurement may not be unnecessary.…”

“…Various methods have been explored to this end, Gupta et al [15] used Long-Term Recurrent Convolutional Network (LRCN), a combination of 2D CNN and LSTM, Abedi and Khan [12] used 2D ResNet along with TCN, Liao et al [13] used Squeeze-and-Excitation ResNet along with LSTM with global attention, and Selim et al [35] used EfficientNet along with LSTM and TCN.…”

“…The results of methods based on functionals are significantly lower than those of sequential and BoS models. 48.10 I3D [16] 52.40 C3D + LSTM [12] 56.60 C3D with transfer learning [15] 57.80 LRCN [15] 57.90 DFSTN [13] 58.80 C3D + TCN [12] 59.90 DERN [21] 60.00 ResNet + LSTM [12] 61.50 Neural Turing Machine [27] 61.30 3D DenseNet [34] 63.60 ResNet + TCN [12] 63.90 ShuffleNet v2 [14] 63.90 EfficientNetB7 + TCN [35] 64.67 Affect-driven Ordinal TCN [37] 67.40 EfficientNetB7 + LSTM [35] 67.…”

Convolutional Neural Network Model based Students’ Engagement Detection in Imbalanced DAiSEE Dataset

“…Selim et al [5] also used the DAiSEE dataset to detect students' engagement and compared the performance of the proposed method with the VRESEE dataset. They proposed a Hybrid EfficientNetB7 model combined with TCN, LSTM, and Bi-LSTM.…”

“…In contrast to previous studies that solely relied on students' facial expressions, they also took into account students' mouse usage behavior in their approach. Bhardwaj et al [4] proposed a deep learning model named Convolutional Neural Network (CNN) for students' engagement detection, while Selim, et al [5] conducted students' engagement detection in online learning using Hybrid EfficientNetB7 together with TCN, LSTM, and Bi-LSTM. Khenkar et al [6] also proposed an engagement detection method based on micro-body gestures using 3D Convolutional Neural Network (CNN).…”