Children’s Play and Problem Solving in Motion-Based Educational Games: Synergies between Human Annotations and Multi-Modal Data

Abstract

Identifying and supporting children's play and problem solving behaviour is important for designing educational technologies. This can inform feedback mechanisms to scaffold learning (provide hints or progress information), and assist facilitators (teachers, parents) in supporting children. Traditionally, researchers manually code video to dissect children's nuanced play and problem solving behaviour. Advancements in sensing technologies and their respective Multi-Modal Data (MMD), afford observation of invisible states (cognitive, affective, physiological), and provide opportunities to inspect internal processes experienced during learning and play. However, limited research combines traditional video annotations and MMD to understand children's behaviour as they interact with educational technology. To address this concern, we collected data from webcam, wristband, eye-trackers, and Kinect, as 26 children, aged 10-12, played a Motion-Based Educational Games (MBEG). Results showed significant differences in children's experience during play and problem solving episodes, and motivate design considerations aimed to facilitate children's interactions with MBEG. CCS CONCEPTS• Human-centered computing → Empirical studies in interaction design; Empirical studies in HCI; • Applied computing → Interactive learning environments.

Wearable Sensing and Quantified-self to explain Learning Experience

“…Finally, we observed in two configurations that students with low levels of stress will be able to achieve high performance, indicating a negative association between the two. The relationship between the stress and learners' performance have been found in related studies with wearable sensor-based data [3], [6]. High stressful situations can be avoided with prompting and supporting the learners to understand the challenges that they are facing and hinting towards proper/correct solutions [3].…”

“…The next three configurations complement each other with high emotional regulation, high engagement and low stress explaining the high perceived performance in a pairwise manner in the three configurations respectively. High engagement often indicates high quality interaction within the learning context and there fore often leads to high performance [3]. Adapting to the learners capability is one of the most sought for research outcome in the related fields such as, Learning Analytics, User Modelling, and Artificial Intelligence in Education.…”

“…Using questionnaires in real-time during learning sessions is rather challenging and would disrupt learners' experience [2]. One way to conduct real-time sensing, while the learners are engaged with their tasks during a learning session, is to use wearable sensors [3]. Several studies have explored the relationship between the physiological data, captured from the wearable devices, and emotions [4], collaboration quality [5] and learning experience [2].…”

“…While multimodal data may offer rich insight, they hinder the pervasiveness of the data collection and raise certain invasive issues (eye-tracking glasses or EEG) or privacy issues (audio or video recordings). On the other hand, a few studies are also based on tedious and error-prone process of human labelling of the collaborative episodes [3], [5]. To overcome the aforementioned issues, we propose to use only the wristband data (to incorporate the sensing in a pervasive manner) and employ completely automatic yet explainable methods to explain the relationships between the physiological data (wristband data in our case) and the learning experiences.…”

Investigating the Effect of Interface Type on Embodied Interactions in 3D Gestural Human-Computer Interaction

“…Through emerging technologies, it is possible to integrate embodiment into interacting with computers in the form of embodied human-computer interaction (eHCI). Embodied interactions have shown promise in several domains (Benford et al, 2021; Dworkin et al, 2018; Lee-Cultura et al, 2020; Sarma & Bhuyan, 2021; Spiel, 2021). Technologies such as augmented and virtual reality implement an eHCI style using virtual reality (VR) headsets or data gloves where people can now use their bodies to interact with computers, but these technologies are invasive to the end user as they are typically bulky equipment.…”