Multimodal Learning Experience for Deliberate Practice

Abstract

While digital education technologies have improved to make educational resources more available, the modes of interaction they implement remain largely unnatural for the learner. Modern sensor-enabled computer systems allow extending human-computer interfaces for multimodal communication. Advances in Artificial Intelligence allow interpreting the data collected from multimodal and multi-sensor devices. These insights can be used to support deliberate practice with personalised feedback and adaptation through Multimodal Learning Experiences (MLX). This chapter elaborates on the approaches, architectures, and methodologies in five different use cases that use multimodal learning analytics applications for deliberate practice.

Beyond hard workout: A multimodal framework for personalised running training with immersive technologies

“…An MLX system can be defined as the digital enhancement of any learning activity that occurs in an authentic learning environment and is analysed with more than two modalities using AI. It relies on three basic principles: the use of sensors, the development of authentic practice and the use of immersive and ubiquitous technological tools (Di Mitri et al., 2022).…”

“…MLX system data are multimodal and can provide contextual, interactional, behavioural and physiological information. In addition, a generic MLX system also has a user interface (interaction layer) whose function is to communicate with the learner and the tutor (Di Mitri et al., 2022).…”

“…The emergence of smart sensing learning technologies led to general approaches such as the multimodal learning experience (MLX). MLX supports deliberate practice of skills based on personalised information using AI to interpret the data collected from multimodal and multisensor technologies (Di Mitri et al., 2022). In this paper, we argue that with the use of sensing and immersive technologies, it is possible to create and design MLX solutions that address the aforementioned drawbacks.…”

Multimodal and immersive systems for skills development and education

“…The combination of multimodal and immersive technologies enables the development of multimodal learning experiences (MLX) (Di Mitri et al., 2022), and learning scenarios that use sensors, AI and feedback mechanisms to enhance deliberate practice. MLX are especially promising for psychomotor skills that require authentic practice in the physical world, such as public speaking (Schneider et al., 2015), cardiopulmonary resuscitation (Di Mitri, Schneider, & Drachsler, 2021), calligraphy (Limbu, Jarodzka, et al., 2019), table tennis (Mat Sanusi et al., 2021) and dance (Geisen et al., 2023).…”

Exploring the role of learning through play in promoting multimodal learning among children: a pilot study in Chinese first-tier cities

“…Human cognition is the sum of multimodal learning (Ivanovic et al, 2018 ). We establish interactions with complex environments through our visual, auditory, and other sensory systems and construct meaning from this external information through the cerebral cortex (Di Mitri et al, 2022 ). Multimodal learning aims to enable children to process and associate information from multiple channels (Schneider et al, 2018 ).…”