Learning linkages: Integrating data streams of multiple modalities and timescales

Abstract

Increasingly, student work is being conducted on computers and online, producing vast amounts of learning‐related data. The educational analytics fields have produced many insights about learning based solely on tutoring systems' automatically logged data, or “log data.” But log data leave out important contextual information about the learning experience. For example, a student working at a computer might be working independently with few outside influences. Alternatively, he or she might be in a lively classroom, with other students around, talking and offering suggestions. Tools that capture these other experiences have potential to augment and complement log data. However, the collection of rich, multimodal data streams and the increased complexity and heterogeneity in the resulting data pose many challenges to researchers. Here, we present two empirical studies that take advantage of multimodal data sources to enrich our understanding of student learning. We leverage and extend quantitative models of student learning to incorporate insights derived jointly from data collected in multiple modalities (log data, video, and high‐fidelity audio) and contexts (individual vs. collaborative classroom learning). We discuss the unique benefits of multimodal data and present methods that take advantage of such benefits while easing the burden on researchers' time and effort.

Multimodal data capabilities for learning: What can multimodal data tell us about learning?

“…Further investigation into how these different modalities were used revealed that logs and videos were mostly used to evaluate the performance of the participants, either in a quantitative way using logs (Liu, Stamper, & Davenport, 2018 ;Liu et al ., 2019 ;Mock et al ., 2016 ;Sharma, Papamitsiou, et al ., 2019 ) or environment variables , or in a qualitative manner using videos (Worsley & Blikstein, 2015, 2018. The rest of the multimodal sources were used to quantify behavioral trajectories, such as interaction behavior using touch gestures (Mock et al ., 2016 ), engagement with problem space using EDA and audio (flow, stress; Worsley & Blikstein, 2015, 2018, understanding and misconceptions using physiological data (Liu et al ., 2018(Liu et al ., , 2019, and problem-solving behavior using faces and EEG (Sharma, Papamitsiou, et al ., 2019 ) and eye-tracking .…”

“…In the same vein, Spikol, Ruffaldi, Dabisias, and Cukurova ( 2018 ) and Blikstein and Worsley ( 2016 ) highlighted the importance and benefits of MMD in open-ended learning tasks, while Noel et al . ( 2018 ) and Liu et al . ( 2019 ) showcased the potential of MMD in rather restricted settings.…”

“…The MMLA studies identified in the learning-task performance category utilized the richness of MMLA to investigate different approaches (eg, making: Worsley & Blikstein, 2018 ;debugging: Mangaroska et al ., 2018 ;playing: Giannakos et al ., 2019 ) with the ultimate goals of understanding the experience (Di Mitri et al ., 2017 ;Giannakos et al ., 2019 ), improving either the technology (eg, or the design of the process (eg, Liu et al ., 2019 ;Worsley & Blikstein, 2018 ) and achieving a higher level of success on the given task. An interesting observation is that all the studies identified in this category were highly contextualized.…”

“…Giannakos, Sharma, Pappas, Kostakos, and Velloso ( 2019 ) found that the prediction of skill acquisition was far better using combined MMD (eg, eye-tracking, Electroencephalography [EEG] and facial video) than when using any individual stream. Research has shown that the fusion of MMD brings significantly better prediction of learning outcomes and helps us to interpret complex learning processes (Giannakos et al ., 2019 ;Liu et al ., 2019 ;Sharma, Papamitsiou, & Giannakos, 2019 ;Sharma, Pappas, Papavlasopoulou, & Giannakos, 2019 ). Despite the great potential of MMD for learning, and recent developments in the context of the CrossMMLA community, research in this direction has not reached its potential.…”

Temporal analysis of multimodal data to predict collaborative learning outcomes

“…Similarly, Sharma et al (2019) showed that a model with gaze, facial and physiological features outperformed unimodal models in an adaptive assessment task. Liu, Stamper, and Davenport (2018), Liu et al (2019) also reported that models involving multiple data streams performed better than log-based models to predict student performance/behavior while solving problems with an ITS. However, the majority of these contributions focused on individual learning contexts.…”