Recognizing Multidimensional Engagement of E-Learners Based on Multi-Channel Data in E-Learning Environment

Abstract

Lack of supervision'' is a particularly challenging problem in E-learning or distance learning environments. A wide range of research efforts and technologies have been explored to alleviate its impact by monitoring students' engagement, such as emotion or learning behaviors. However, the current research still lacks multi-dimensional computational measures for analyzing learner's engagement from the interactions that occur in digital learning environment. In this paper, we propose an integrated framework to identify learning engagement from three facets: affect, behavior and cognitive state, which are conveyed by learner's facial expressions, eye movement behaviors and the overall performance during short video learning session. To recognize the three states of learners, three channel data is recorded: 1) video/image sequence captured by camera; 2) eye movement information from a non-intrusive and cost-effective eye tracker; and 3) click stream data from mouse. Based on these modalities, a multi-channel data fusion strategy is designed that concatenates time series features of three channels in the same time segment to predict course learning performance. We also presented a new method to make the self-reported annotations more reliable without using external observers' verification. To validate the approach and methods, 46 participants were invited to attend a representative on-line course that consists of short videos in our designed learning environment. The results demonstrated the effectiveness of the proposed framework and methods in monitoring learning engagement. More importantly, a prototype system was developed to detect learner's emotional and eye behavioral engagement in real-time as well as predict the learning performance of learners after they had completed each short video course.INDEX TERMS E-learning, engagement recognition, multi-channel data fusion, learning performance prediction.

A review on data fusion in multimodal learning analytics and educational data mining

“…In contrast, N. L. Henderson, Rowe, Mott, and Lester (2019), reduced the dimensionality of the features using principal component analysis (PCA) in two different configurations: (a) they concatenated all of the features of the sources and applied PCA to the resulting vector; (b) they applied PCA to the features of each source first and concatenated the results following the reduction of dimensionality. Yue et al (2019) selected the best features first and then reduced dimensionality using two approaches, PCA and a Kolmogorov–Smirnov test. The other studies in the early category based fusion on mere concatenation of the features extracted from each source into a single vector of features which fed into the subsequent analysis.…”

“…The study by Liu et al (2019) had a wide range of different types of data with numerous means for capturing it, whereas Nam Liao et al (2019) went in the opposite direction, as it only included numerical digital data. Yue et al (2019) used a wide range of fused data (facial expressions, eye tracking, grades, etc.) and was the only study to include an open data source in the fusion.…”

“…L. Henderson, Rowe, Mott, Brawner, et al, 2019;N. Henderson et al, 2020;Liu et al, 2019;Nam Liao et al, 2019;Olsen et al, 2020;Peng & Nagao, 2021;Prieto et al, 2018;Shankar et al, 2019;Worsley, 2014;Wu et al, 2020;Xu et al, 2019;Yue et al, 2019;Di Mitri et al, 2017;Sharma et al, 2019;Hussain et al, 2011 Ensembles Applies the idea of ensembles (machine learning) to combine the data from the various classifiers' decisions Chango, Cerezo, Sanchez-Santillan, et al, 2021;Wu et al, 2020 Statistical operators…”

Current and future of technologies and services in smart e-learning

“…Smart e-learning refers to a new paradigm of online education informatization, based on educational data, to integrate modern education theory with supporting technologies to achieve In smart e-learning, descriptive analytics aims to describe, summarize, and analyse historical and educational data and carry out typical tasks that include e-learner profiling [2] and KG construction [5]. Diagnostic analytics is to identify causes of learning trends and outcomes of e-learners, while continuous analytics is to monitor the status of e-learners and resource utilization, decide, and act autonomously or semi-autonomously.…”

The Use of Special-Purpose Simulators as an Advanced Professional Training Tool Under E-Learning Conditions

“…E-learning has become a generalized strategy that has helped in a short time to make the transition to a new training model that has resulted in greater interaction with concepts, more student dedication, a higher level of commitment, and better assimilation (Mystakidis, Berki, and Valtanen, 2019). However, it is also true that it has produced deficiencies at an emotional level in terms of social interaction between individuals and has neglected the practical aspects of laboratory and on-site training that are fundamental to technical and technological training courses (Baki, Birgoren, and Aktepe, 2018;You and Robert, 2018;Yue et al, 2019). From these weaknesses of the new model arises the need for rapid update and improvement that allows the level of interaction of laboratories while strengthening the positive elements of Elearning and observing the rules of social isolation (Flor, Belmonte, and Fabregat, 2018; Lin, Wang, Wu, and Chen, 2019).…”