Multimodal data capabilities for learning: What can multimodal data tell us about learning?

Abstract

Most research on learning technology uses clickstreams and questionnaires as their primary source of quantitative data. This study presents the outcomes of a systematic literature review of empirical evidence on the capabilities of multimodal data (MMD) for human learning. This paper provides an overview of what and how MMD have been used to inform learning and in what contexts. A search resulted in 42 papers that were included in the analysis. The results of the review depict the capabilities of MMD for learning and the ongoing advances and implications that emerge from the employment of MMD to capture and improve learning. In particular, we identified the six main objectives (ie, behavioral trajectories, learning outcome, learning‐task performance, teacher support, engagement and student feedback) that the MMLA research has been focusing on. We also summarize the implications derived from the reviewed articles and frame them within six thematic areas. Finally, this review stresses that future research should consider developing a framework that would enable MMD capacities to be aligned with the research and learning design (LD). These MMD capacities could also be utilized on furthering theory and practice. Our findings set a baseline to support the adoption and democratization of MMD within future learning technology research and development. What is already known about this topic Capturing and measuring learners’ engagement and behavior using MMD has been explored in recent years and exhibits great potential. There are documented challenges and opportunities associated with capturing, processing, analyzing and interpreting MMD to support human learning. MMD can provide insights into predicting learning engagement and performance as well as into supporting the process. What this paper adds Provides a systematic literature review (SLR) of empirical evidence on MMD for human learning. Summarizes the insights MMD can give us about the learning outcomes and process. Identifies challenges and opportunities of MMD to support human learning. Implications for practice and/or policy Learning analytics researchers will be able to use the SLR as a guide for future research. Learning analytics practitioners will be able to use the SLR as a summary of the current state of the field.

The promise and challenges of multimodal learning analytics

“…To address this, MMLA research focuses on the use of physical sensors, due to their capacity to capture physiological and behavioural manifestations of learners’ emotions and traits that cannot be easily seen by the human naked eye such as learners’ emotions, gaze, cognitive states and bodily responses. For instance, recent work in MMLA has enabled performing text, speech, handwriting, sketch, gesture, affect, or eye‐gaze analysis (eg, Blikstein & Worsley, 2016; Sharma & Giannakos, 2020; Spikol, Ruffaldi, Dabisias, & Cukurova, 2018), in order to extract indicators and use these to model higher‐order cognitive, meta‐cognitive and affective factors for research purposes or to provide automated feedback to enable learner reflections (eg, Ochoa et al ., 2018). As it appears in this special issue, Sharma and Giannakos (2020) summarise the state‐of‐the‐art in various learning capacities of multimodal data as well as the critical insights about MMLA in the form of a systematic literature review.…”

“…That is also portrayed with the formation of the CrossMMLA SIG and the continuous growth of its respective workshops at LAK conferences (ie, Martinez‐Maldonado et al ., 2018; Spikol et al ., 2017). The contributions in this special issue cover several dimensions of these MMLA research trends, as also identified from the recent literature review of the field included in this issue (Sharma, & Giannakos, 2020). We expect and welcome further work on these dimensions and would like to conclude this editorial piece reflecting upon three main themes of challenges in MMLA research.…”

Adult learning in online communities of practice: A systematic review

“…The process of collecting published studies of acceptable quality is a methodical and meticulous one. A systematic criteria need to be used for selection to reduce researcher bias and provide transparency (Bano et al., 2018; Sharma & Giannakos, 2020). We used the guidelines provided by Kitchenham and Charters (2007) to conduct our systematic review.…”

“…The selection phase determines the overall validity of the studies for the literature review, which is why it is important to define specific inclusion and exclusion criteria. We applied four quality criteria (see Appendix 2) informed by related works (Sharma & Giannakos, 2020). Therefore, studies were eligible for inclusion if they were focused on adult learning in OCOPs.…”

Multimodal Technologies in Precision Education: Providing New Opportunities or Adding More Challenges?

“…Upon completion of this process a total of 45 articles met the inclusion criteria. The manuscripts selected for detailed analysis and data extraction were divided across the readers in accordance with their field of expertise, which was carried out partially following the data extraction rubric adapted by Sharma and Giannakos [35]. The findings reported in the forthcoming sections constitute a synthesis of the research efforts that have been identified with a direct interest in facilitating PE and have been further enriched with distilled ideas for practice, policy and further research through reflection and multidisciplinary discourse.…”