Three-dimensional DenseNet self-attention neural network for automatic detection of student’s engagement

Abstract

Today, due to the widespread outbreak of the deadly coronavirus, popularly known as COVID-19, the traditional classroom education has been shifted to computer-based learning. Students of various cognitive and psychological abilities participate in the learning process. However, most students are hesitant to provide regular and honest feedback on the comprehensiveness of the course, making it difficult for the instructor to ensure that all students are grasping the information at the same rate. The students’ understanding of the course and their emotional engagement, as indicated via facial expressions, are intertwined. This paper attempts to present a three-dimensional DenseNet self-attention neural network (DenseAttNet) used to identify and evaluate student participation in modern and traditional educational programs. With the Dataset for Affective States in E-Environments (DAiSEE), the proposed DenseAttNet model outperformed all other existing methods, achieving baseline accuracy of 63.59% for engagement classification and 54.27% for boredom classification, respectively. Besides, DenseAttNet trained on all four multi-labels, namely boredom, engagement, confusion, and frustration has registered an accuracy of 81.17%, 94.85%, 90.96%, and 95.85%, respectively. In addition, we performed a regression experiment on DAiSEE and obtained the lowest Mean Square Error (MSE) value of 0.0347. Finally, the proposed approach achieves a competitive MSE of 0.0877 when validated on the Emotion Recognition in the Wild Engagement Prediction (EmotiW-EP) dataset.

Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data

“…The input data of the network are the paired MRI and EHR data. 3D ResNet-50 ( Yu et al, 2021 ; Mehta et al, 2022 ) is the backbone network in the initial stage. Other deeper networks, such as DenseNet ( Huang et al, 2017 ), also work with our proposed framework.…”

“…Attention can provide the ability to highlight vital information and suppress irrelevant information. In the tasks of medical imaging analysis, the spatial–temporal self-attention mechanism ( Schlemper et al, 2019 ; Chen and Shi, 2020 ; Chen et al, 2020 ; Plizzari et al, 2021 ; Yu et al, 2021 ; Mehta et al, 2022 ) is often used to capture the spatial and temporal correlations of the same image sequences. The cross-attention mechanism ( Hou et al, 2019 ; Huang et al, 2019 ; Yu et al, 2021 ) can capture the interdependent relationship between two sequences of single-modal or multimodal data by integrating two separate embedding sequences with the same dimension asymmetrically.…”

Students Engagement Level Detection in Online e-Learning Using Hybrid EfficientNetB7 Together With TCN, LSTM, and Bi-LSTM

“…The accuracy achieved by DERN [12] was 60%, which is higher than DFSTN and C3D + TCN [11] by 1.16% and 0.3% respectively. The 3D DenseAttNet achieved 63.59% [9]. Finally, [11] introduced ResNet + TCN with sampling and weighted loss, C3D + TCN, ResNet + LSTM and ResNet + TCN models, which reached accuracy of 53.7%, 59.97%, 61.15% and 63.9% respectively.…”

“…Abedi et al [11] improved student engagement recognition using the end-to-end Residual Network (ResNet) model, followed by Temporal Convolutional Network (TCN), reaching 63.9% accuracy on the DAiSEE dataset in four-level classification. Mehta et al [9] proposed a three-dimensional DenseNet Self-Attention neural network (3D DenseAttNet) for students' engagement detection in online learning, achieving an accuracy of 63.59% in four-level classification.…”

“…Model Accuracy video-level InceptionNet [10] 46.40% frame-level InceptionNet [10] 47.10% C3D feature extraction [10] 67.99% I3D [37] 52.35% ResNet+TCN with weighted loss [11] 53.7% C3D fine tuning [10] 56.10% C3D (focal loss) [13] 56.20% LRCN [10] 57.90% DFSTN [14] 58.84% C3D + TCN [11] 59.97% DERN [12] 60% ResNet + LSTM [11] 61.15% ResNet + TCN [11] 63.9% 3D DenseAttNet [9] 63.59% proposed hybrid EfficientNet B7 + TCN 64.67% proposed hybrid EfficientNet B7 + Bi-LSTM 66.39% proposed hybrid EfficientNet B7 + LSTM 67.48%…”