Recognizing Student Emotions using Brainwaves and Mouse Behavior Data

Abstract

Brainwaves (EEG signals) and mouse behavior information are shown to be useful in predicting academic emotions, such as confidence, excitement, frustration and interest. Twenty five college students were asked to use the Aplusix math learning software while their brainwaves signals and mouse behavior (number of clicks, duration of each click, distance traveled by the mouse) were automatically being captured. It is shown that by combining the extracted features from EEG signals with data representing mouse click behavior, the accuracy in predicting academic emotions substantially increases compared to using only features extracted from EEG signals or just mouse behavior alone. Furthermore, experiments were conducted to assess the prediction accuracy of the system at points during the learning session where several of the extracted features significantly deviate in value from their mean. The experiments confirm that the prediction performance increases as the number of feature values that deviate significantly from the mean increases.

On the joint use of Artificial Intelligence and Brain-Imaging Techniques in Technology-enhanced Learning Environments: A Systematic Literature Review

“…Some studies investigated the combination of data retrieved from brain-imaging techniques with data retrieved from other devices to produce more robust data sets in order to better classify students' cognitive phenomena in the technology-enhanced learning context. For example, studies that aimed to detect students' emotions combined data related to students' brain activities with data related to facial recognition (retrieved with camera (Zatarain Cabada et al, 2019)) and related to behavior in the system (retrieved with mouse (Azcarraga & Suarez, 2013)).…”

“…Concerning the classification of students' emotions, Gruenewald et al (2018) reported an F-score of 91.49% using Support vector machine -SVM with RBF kernel, as seen in Table 3. In Azcarraga & Suarez (2013), Multilayer Perceptron -MLP presented an F-score of 88% to classify students' emotions.…”

“…A mean accuracy rate of 91% across all the participants was reached. Azcarraga & Suarez (2013) used a multi-layered perceptron (MLP) and support vector machine (SVM) to predict academic emotions (confidence, excitement, frustration, and interest) during learning sessions. The data used was collected from EEG devices and mouse click information while students were solving algebra equations.…”

Emotion theory in education research practice: an interdisciplinary critical literature review

“…However, while molecular and genetic explanations of cognitive variance such as the COMT literature have high predictive power and may be valuable in making public policy decisions which are equitable for diverse neurotypes (this is the accepted spelling in the neurodiversity advocacy community), they offer little data for assessing one learning system, or comparing one system versus another, because they do not measure temporal shifts in emotion during learning. Azcarraga and Suarez (2013) report some preliminary findings on the use of EEG signals and pressure-sensitive mouse behaviors in college-age learning. This type of data is more promising on the temporal precision front, but at this time the authors' rates of correlation and p-values remain marginal.…”

A Survey of AI-Based Facial Emotion Recognition: Features, ML & DL Techniques, Age-Wise Datasets and Future Directions

“…So, it is essential to explore this research domain in more detail as less accurate systems plague its commercial implementation. Human facial emotion recognition has been broadly used in numerous human-computer interactions such as smartphones, affective computing, intelligent control systems, psychological, behavioral study, pattern searching, defense, social sites, robotics, and other fields [3][4][5]. By evaluating these emotions, one could deliver maximum user satisfaction and feedback to improve current technologies.…”