Motion-Based Educational Games: Using Multi-Modal Data to Predict Player’s Performance

Abstract

Multi-Modal Data (MMD) can help educational games researchers understand the synergistic relationship between player's movement and their learning experiences, and consequently uncover insights that may lead to improved design of movement-based game technologies for learning. Predicting player performance fosters opportunities to cultivate heightened educational experiences and outcomes. However, predicting player's performance utilising player-generated MMD during their interactions with educational Motion-Based Touchless Games (MBTG) is challenging. To bridge this gap, we implemented an in-situ study where 26 users, age 11, played 2 maths MBTGs in a single 20-30 minute session. We collected player's MMD (i.e., gaze data from eye-tracking glasses, physiological data from wristbands, and skeleton data from Kinect) produced during game-play. To investigate the potential of MMD for predicting player's academic performance, we used machine learning techniques and MMD derived from player's game-play. This allowed us to identify the MMD features that drive rapid highly accurate predictions of players' academic performance in educational MBTGs. This might allow us to provide realtime proactive feedback to the player to support them through their educational gaming experience. Our analysis compared two data lengths corresponding to half and full duration of the player's question solving time. We showed that all combinations of extracted features associated with gaze, physiological, and skeleton data, predicted student performance more accurately than the majority baseline. Additionally, the most accurate prediction of player's performance derived from the combination of gaze and physiological data for both full and half data lengths. Our findings emphasise the significance of using MMD for realtime performance prediction in educational MBTG and offer implications for practice.

Keep Calm and Do Not Carry-Forward: Toward Sensor-Data Driven AI Agent to Enhance Human Learning

“…In numerous studies, predictive performance models containing fused data sources have outperformed predictive performance of the individual data sources (Cukurova et al, 2019 ; Giannakos et al, 2019 ; Liu et al, 2019 ; Sharma et al, 2019b , 2020c ; Lee-Cultura et al, 2020a ). For example, one study which used a modified-Pacman game, found that the fusion of EEG, eye-tracking and facial data streams, outperformed the individual data streams when predicting player performance (Giannakos et al, 2019 ).…”

“…Similarly, in an adaptive self-assessment test, the combination of eye-tracking, facial features, EDA, and HRV data showed lower error rates than the individual components when predicting engagement and performance (Sharma et al, 2019b ). In the same vein, combining features from eye-tracking, motion, EDA and HRV have resulted in better performance prediction during children's play with MBEG, than the individual data stream (Lee-Cultura et al, 2020a ). Lastly, in a diverse set of studies (e.g., games and learning tasks), the combination of facial data, EDA, BVP, and HRV resulted in a lower error rate while predicting participant's cognitive performance when compared against the error rate achieved by the individual features (Sharma et al, 2020c ).…”

“…These prediction efforts involve longitudinally collected data with similar long term dependent variables (e.g., dropout, at-risk students, low grades). To the best of our knowledge, there are limited short-term studies on early prediction (Lee-Cultura et al, 2020a ), and other efforts are in a very specific context of intelligent tutoring system (Piech et al, 2015 ; Chen et al, 2018 ; Bhatt et al, 2020 ). Moreover, several studies employ methods that are based on complex algorithms which are difficult to describe (e.g., deep learning and hidden Markov models) and, thus, act as a “black-box” prediction of performance/engagement (aside a small number which use open learner models Badea and Popescu, 2020 ; Hooshyar et al, 2020 ; Somyürek et al, 2020 ).…”

Characterising Players of a Cube Puzzle Game with a Two-level Bag of Words

“…Player experience (challenge, frustration, and fun) can be modelled through controllable features of level design [21,22]. Instead of having users report their personality or emotions explicitly during the game play, it is interesting to do this unsupervisedly and implicitly [4], or through gaze, and physiological data [19]. Sequential models of in-game player behaviours are an alternative to aggregated players' actions, for predicting personality and expertise [6], assistance in serious games [36], churn prediction [17,38], or player categorisation from past and predicted behaviours [8].…”

Children’s Play and Problem Solving in Motion-Based Educational Games: Synergies between Human Annotations and Multi-Modal Data

“…Such data collections empower us to transcend the limits of human observation, by accessing real-time information on children's seemingly "invisible" cognitive, affective and physiological states [92]. Accordingly, sensing technologies are gaining traction as useful, reliable means of investigative practice for understanding multi-faceted problem solving phenomena and supporting learning in-situ [11,19], specifically in the domain of children's problem solving behaviours during interactive learning experiences [33,51,52,71]. Additionally, sensing technologies and their respective MMD, allow us to closely monitor and understand children's play and problem solving behaviours, leveraging the key affordances of MMD (e.g., temporality and direct access to indicators of children's cognitive and affective processes [19]).…”