Keep Calm and Do Not Carry-Forward: Toward Sensor-Data Driven AI Agent to Enhance Human Learning

Abstract

The integration of Multimodal Data (MMD) and embodied learning systems (such as Motion Based Educational Games, MBEG), can help learning researchers to better understand the synergy between students' interactions and their learning experiences. Unfolding the dynamics behind this important synergy can lead to the design of intelligent agents which leverage students' movements and support their learning. However, real-time use of student-generated MMD derived from their interactions with embodied learning systems (MBEG in our case) is challenging and remains under-explored due to its complexity (e.g., handle sensor-data and enable an AI agent to use them). To bridge this gap, we conducted an in-situ study where 40 children, aged 9–12, played MBEG on maths and language development. We automatically, unobtrusively, and continuously monitored students' experiences using eye-tracking glasses, physiological wristbands, and Kinect, during game-play. This allowed us to understand the different cognitive and physiological dimensions of students' progress (right/wrong responses) during the three different stages of the MBEG problem-solving processes, namely the “see-solve-move-respond” (S2MR) cycle. We introduce the novel Carry Forward Effect (CFE); a phenomenon occurring in such games, whereby students propagate, or “carry forward,” the cognitive and physiological effects derived from their MMD, to subsequent phases in the see-solve-move-respond cycle. By identifying moments when the Carry Forward Effect is congruent (or not) to students' learning performance, we uncover opportunities for feedback delivery to encourage or subdue the impact of the CFE. Our results demonstrate the importance of wristband and eye-tracking data as key indicators for prioritizing adaptive feedback to support students in MBEG and emphasize the significance of using MMD to support students' performance in real-time educational settings.

The Impact of Emotional Feedback and Elaborated Feedback of a Pedagogical Agent on Multimedia Learning

“…To gain more objective evidence and holistic understanding, future work should use multimodal data (the fusion of information extracted from multiple data sources, such as eye-tracking, EEG, facial data streams, physiological indexes, etc.). For example, data sources from self-report-scale and eye-tracking during learners' interaction with a learning system can be integrated and used to measure cognitive load (Sharma et al, 2022 ).…”

Artificial intelligence in lean manufacturing: digitalization with a human touch?

“…Though more general research has investigated the role of AI in enhancing human learning (Sharma et al , 2022), for example, how generative AI might accelerate human learning (Johnson, 2023), manufacturing-specific examples are lacking. Just as Sakichi Toyoda developed Jidoka over 100 years ago, introducing a form of intelligence to mechanical automation to drive improvement and learning throughout the firm, the future developments of AI in lean manufacturing should consider how such technology can be used to promote learning and improvement within and across organizations.…”

“…Given the emergent nature of the research topic, rapid literature review (RLR) was selected to investigate the current and future application areas for AI in lean manufacturing. RLR is an alternative to systematic literature review (SLR) that can speed up the analysis of newly published data (Smela et al, 2023). To ensure that the search process was effective and efficient in producing results in a timely manner, only the research database Scopus was used (according to Bjørnbet et al (2021), Scopus covers a satisfactory share of relevant, extant literature and produces less noise, compared to other databases).…”

Multisensory Interaction and Analytics to Enhance Smart Learning Environments: A Systematic Literature Review

“…This learnercentered approach raises further discussion of how LA and MMLA might aid in the creation of customized multisensory learning experiences. Although such data can enable the use of automated feedback (e.g., errors correction, hints, or feedback provision) [94] and adaptations to support effective combinations of stimuli and interaction [95], [96], [97], [98], this potential remains unexplored in MSEs.…”