Using Multimodal Learning Analytics to Model Student Behavior: A Systematic Analysis of Epistemological Framing

Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques. What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential. Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome. Multimodal data can accurately predict learning engagement, performance and processes. What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach. Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings. Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity. Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context. Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states. The constraints from th...

Section: Related Work: Utilizing Multimodal Data To Predict Learning mentioning

confidence: 99%

Section: Related Work: Utilizing Multimodal Data To Predict Learning mentioning

confidence: 99%

See 1 more Smart Citation

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Sharma¹,

Papamitsiou²,

Giannakos³

2019

“…The confluence of multimodal data (MMD) with advanced computational analyses (multimodal learning analytics-MMLA, as the literature refers to them) enables us to understand and support complex learning phenomena (Blikstein & Worsley, 2016 ). For example, eye-tracking data and the different linguistic and prosodic features of speech can inform us about the students' expertise (Andrade, Delandshere, & Danish, 2016 ;Mangaroska, Vesin, & Giannakos, 2019 ); or video data can tell us about their engagement (Nguyen, Huptych, & Rienties, 2018 ;Pardo, Han, & Ellis, 2016 ). These insights can enable actionable feedback to be provided to the learners.…”

Section: Introductionmentioning

confidence: 99%

Multimodal data capabilities for learning: What can multimodal data tell us about learning?

Sharma¹,

Giannakos²

2020

112

Most research on learning technology uses clickstreams and questionnaires as their primary source of quantitative data. This study presents the outcomes of a systematic literature review of empirical evidence on the capabilities of multimodal data (MMD) for human learning. This paper provides an overview of what and how MMD have been used to inform learning and in what contexts. A search resulted in 42 papers that were included in the analysis. The results of the review depict the capabilities of MMD for learning and the ongoing advances and implications that emerge from the employment of MMD to capture and improve learning. In particular, we identified the six main objectives (ie, behavioral trajectories, learning outcome, learning‐task performance, teacher support, engagement and student feedback) that the MMLA research has been focusing on. We also summarize the implications derived from the reviewed articles and frame them within six thematic areas. Finally, this review stresses that future research should consider developing a framework that would enable MMD capacities to be aligned with the research and learning design (LD). These MMD capacities could also be utilized on furthering theory and practice. Our findings set a baseline to support the adoption and democratization of MMD within future learning technology research and development. What is already known about this topic Capturing and measuring learners’ engagement and behavior using MMD has been explored in recent years and exhibits great potential. There are documented challenges and opportunities associated with capturing, processing, analyzing and interpreting MMD to support human learning. MMD can provide insights into predicting learning engagement and performance as well as into supporting the process. What this paper adds Provides a systematic literature review (SLR) of empirical evidence on MMD for human learning. Summarizes the insights MMD can give us about the learning outcomes and process. Identifies challenges and opportunities of MMD to support human learning. Implications for practice and/or policy Learning analytics researchers will be able to use the SLR as a guide for future research. Learning analytics practitioners will be able to use the SLR as a summary of the current state of the field.

“…Again using hidden Markov models and sequential analysis methods, reasoning patterns within the learning process have been discovered (Andrade, 2017;Chen & Yeh, 2017;Ozturk, Deryakulu, Ozcinar, & Atal, 2014). Researchers have also applied hidden Markov models to multimodal data to detect students' engagement (Andrade et al, 2016). In collaborative settings, sequential analysis, statistical discourse analysis and recurrence analysis have provided insights into the collaborative process (Andrist, Ruis, & Shaffer, 2018;Chiu & Fujita, 2014;Dindar, Alikhani, Malmberg, Järvelä, & Seppänen, 2019;Jermann & Nüssli, 2012;Lee & Tan, 2017;Malmberg, Järvelä, & Järvenoja, 2017).…”

Section: Temporal Analysis In Educationmentioning

confidence: 99%

Temporal analysis of multimodal data to predict collaborative learning outcomes

Olsen¹,

Sharma²,

Rummel³

et al. 2020

The analysis of multiple data streams is a long‐standing practice within educational research. Both multimodal data analysis and temporal analysis have been applied successfully, but in the area of collaborative learning, very few studies have investigated specific advantages of multiple modalities versus a single modality, especially combined with temporal analysis. In this paper, we investigate how both the use of multimodal data and moving from averages and counts to temporal aspects in a collaborative setting provides a better prediction of learning gains. To address these questions, we analyze multimodal data collected from 25 9–11‐year‐old dyads using a fractions intelligent tutoring system. Assessing the relation of dual gaze, tutor log, audio and dialog data to students' learning gains, we find that a combination of modalities, especially those at a smaller time scale, such as gaze and audio, provides a more accurate prediction of learning gains than models with a single modality. Our work contributes to the understanding of how analyzing multimodal data in temporal manner provides additional information around the collaborative learning process.