The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review

Abstract

There is a growing interest in the research and use of multimodal data in learning analytics. This paper presents a systematic literature review of multimodal learning analytics (MMLA) research to assess i) the available evidence of impact on learning outcomes in real-world contexts and ii) explore the extent to which ethical considerations are addressed. A few recent literature reviews argue for the promising value of multimodal data in learning analytics research. However, our understanding of the challenges associated with MMLA research from real-world teaching and learning environments is limited. To address this gap, this paper provides an overview of the evidence of impact and ethical considerations stemming from an analysis of the relevant MMLA research published in the last decade. The search of the literature resulted in 663 papers, of which 100 were included in the final synthesis. The results show that the evidence of real-world impact on learning outcomes is weak, and ethical aspects of MMLA work are rarely addressed. We discuss our results through the lenses of two theoretical frameworks for evidence of impact types and ethical dimensions of MMLA. We conclude that for MMLA to stay relevant and become part of mainstream education, future research should directly address the gaps identified in this review.

Measuring and classifying students' cognitive load in pen‐based mobile learning using handwriting, touch gestural and eye‐tracking data

“…As a result, they may bring in more ethical challenges. Alwahaby et al (2022) summarized several key ethical considerations that should be given when implementing MMLA, such as transparency, accountability, privacy, fairness and bias. Particularly, the process of collection, modelling and visualization of eye-tracking data should be transparent.…”

The role of learning theory in multimodal learning analytics

“…To ensure a high‐quality literature review on the role of learning theory in MMLA research, we first identified the corpus. The selection of papers was primarily based on recent systematic MMLA literature reviews by Sharma and Giannakos (2020) and Alwahaby et al (2022a) for 2010–20 and an additional search by the authors. In particular, we manually reviewed the papers highlighted by Sharma and Giannakos (2020) 2 and Alwahaby et al (2022a) 3 and identified 28 papers from both that explicitly address theoretical positions (13 and 15 papers, respectively).…”

“…The selection of papers was primarily based on recent systematic MMLA literature reviews by Sharma and Giannakos (2020) and Alwahaby et al (2022a) for 2010–20 and an additional search by the authors. In particular, we manually reviewed the papers highlighted by Sharma and Giannakos (2020) 2 and Alwahaby et al (2022a) 3 and identified 28 papers from both that explicitly address theoretical positions (13 and 15 papers, respectively). Moreover, we manually checked the proceedings of LAK 2021–22 and recent special issues by Abrahamson et al (2022), Azevedo and Gašević (2019) and Cukurova et al (2020).…”

“…Previous reviews have described cases where MMLA can enrich our understanding beyond what can be done with mainstream LA (Blikstein & Worsley, 2016) or provide a conceptual framework using handpicked studies (Di Mitri et al, 2018). More recent works have also systematically reviewed the state of the MMLA field but with different foci, including a focus on MMLA capabilities (Sharma & Giannakos, 2020), on the evidence of impact and ethical considerations of MMLA research and practice (Alwahaby et al, 2022a), on the methodological developments in the field (Chua et al, 2019), and on MMLA research with children (Crescenzi‐Lanna, 2020). Despite the importance of these review studies, they have limited focus (or none at all) on the role of theory.…”

From Sensor Data to Educational Insights

“…However, this increased interest in using sensor data in education comes with some important caveats. The risks of unintended over-surveillance and the capture of activities that are not directly related to learning need to be addressed [ 20 ]. For example, using video in learning spaces can bring numerous ethical implications since not all the activities that occur in a classroom may be related to the subject matter being learned, and students can be easily identified [ 21 ].…”