Detecting Academic Affective States of Learners in Online Learning Environments Using Deep Transfer Learning

Abstract

Online Learning Environments (OLEs) have become essential in global education, especially during and after the COVID-19 pandemic. However, OLEs face a challenge in recognizing student emotions, hindering educators' ability to provide effective support. To address this issue, researchers emphasize the importance of a balanced dataset and a precise model for academic emotion detection in OLEs. However, the widely-used DAiSEE dataset is imbalanced and contains videos captured in well-lit environments. However, real-time observations reveal students' diverse lighting conditions and proximity to cameras. Consequently, models trained on DAiSEE dataset exhibit poor accuracy. In response, this work suggests a customized DAiSEE dataset and proposes the Xception-based transfer learned model and AffectXception model. Our customization process involves selectively extracting single-label frames with intensity levels 2 or 3 from the original DAiSEE dataset. To enhance dataset diversity and tackle the issue of dataset imbalance, we meticulously apply data augmentation techniques on these extracted frames. This results in frames that showcase variations in lighting, both low and high, as well as diverse camera perspectives. As a result, the customized DAiSEE dataset is now well-balanced and exceptionally suitable for training deep learning models to detect academic emotions in online learners. Then we trained and tested both proposed models on this dataset. The AffectXception model outperforms existing models, achieving significant improvements. For Boredom, Engagement, Confusion, and Frustration, it attains accuracy rates of 77%, 79.28%, 83.76%, and 91.87%, respectively. Additionally, we evaluate the AffectXception model on the Online Learning Spontaneous Facial Expression Database (OL-SFED), obtaining competitive results across various emotion classes. This work empowers educators to adjust their content and delivery methods based on learners' emotional states, resulting in more effective and informative online sessions. As OLEs continue to play a crucial role in education, our approach enhances their capacity to address students' emotional needs.