Measuring and classifying students' cognitive load in pen‐based mobile learning using handwriting, touch gestural and eye‐tracking data

Abstract

Although the utilization of mobile technologies has recently emerged in various educational settings, limited research has focused on cognitive load detection in the pen‐based learning process. This research conducted two experimental studies to investigate what and how multimodal data can be used to measure and classify learners' real‐time cognitive load. The results found that it was a promising method to predict learners' cognitive load by analysing their handwriting, touch gestural and eye‐tracking data individually and conjunctively. The machine learning approach used in this research achieved a prediction accuracy of 0.86 area under the receiver operating characteristic curve (AUC) and 0.85/0.86 sensitivity/specificity by only using handwriting data, 0.93 AUC and 0.93/0.94 sensitivity/specificity by only using touch gestural data, and 0.94 AUC and 0.94/0.95 sensitivity/specificity by using both the touch gestural and eye‐tracking data. The results can contribute to the optimization of cognitive load and the development of adaptive learning systems for pen‐based mobile learning. Practitioner notesWhat is already known about this topic Pen‐based mobile learning systems allow natural ways of handwriting and gestural touching, which can facilitate learners' cognitive processes in mobile learning. Behavioural and physiological multimodal data are helpful in detecting learners' real‐time cognitive load in mobile learning. The effectiveness of behavioural and physiological multimodal data for measuring cognitive load in pen‐based mobile learning is limited investigated. What this paper adds This paper confirms the effectiveness of handwriting and touch gestural multimodal data for measuring pen‐based learning cognitive load, in terms of their stroke‐, path‐ and time‐based features. This paper explores the potential of eye‐tracking data in measuring pen‐based learning cognitive load. A combination of behavioural and physiological multimodal data is reported to increase the prediction accuracy for cognitive load measurement. Implications for practice and/or policy Practitioners are suggested to use behavioural and physiological multimodal data individually or conjunctively for measuring cognitive load in pen‐based learning. The results provide guides for developing adaptive pen‐based learning systems by optimizing the real‐time cognitive load.