Multimodal data indicators for capturing cognitive, motivational, and emotional learning processes: A systematic literature review

Abstract

This systematic review on data modalities synthesises the research findings in terms of how to optimally use and combine such modalities when investigating cognitive, motivational, and emotional learning processes. ERIC, WoS, and ScienceDirect databases were searched with specific keywords and inclusion criteria for research on data modalities, resulting in 207 relevant publications. We provide findings in terms of target journal, country, subject, participant characteristics, educational level, foci, type of data modality, research method, type of learning, learning setting, and modalities used to study the different foci. In total, 18 data modalities were classified. For the 207 multimodal publications, 721 occurrences of modalities were observed. The most popular modality was interview followed by survey and observation. The least common modalities were heart rate variability, facial expression recognition, and screen recording. From the 207 publications, 98 focused exclusively on the cognitive aspects of learning, followed by 27 publications that only focused on motivation, while only five publications exclusively focused on emotional aspects. Only 10 publications focused on a combination of cognitive, motivational, and emotional aspects of learning. Our results plea for the increased use of objective measures, highlight the need for triangulation of objective and subjective data, and demand for more research on combining various aspects of learning. Further, rather than researching cognitive, motivational, and emotional aspects of learning separately, we encourage scholars to tap into multiple learning processes with multimodal data to derive a more comprehensive view on the phenomenon of learning.

Enriching problem-solving followed by instruction with explanatory accounts of emotions

“…More broadly, despite an increasing number of multimodal research studies being conducted in university settings with undergraduate students as the target group (from 2000-2017), the dominant data collection modality comprises interviews, surveys, and audio/video observations focused primarily on measuring the cognitive aspect of learning. For instance, only 11% of all multimodal research studies included in a recent review of the field studied the emotional aspects of learning (Noroozi et al, 2020).…”

Detecting Emotions through Electrodermal Activity in Learning Contexts: A Systematic Review

“…The multimodal approach examines the relations between physiological arousal as measured by EDA and multimodal data streams to provide insights into the learning process. This approach is in line with the increased focus on the value of data to further understand learning in the field of multimodal learning analytics with specific attention to processes during learning [ 44 , 45 ]. The multi-componential nature of emotion points to individual differences in expression and experience of these emotions [ 6 , 46 ].…”

“…The second strand examines the interaction between physiological arousal as measured by EDA and learning processes. There is a clear trend of studies that specifically focus on what happens during the learning process visible in educational research [ 44 , 45 ]. EDA is viewed as one of multiple data streams, which results in both unimodal and multimodal approaches.…”

Quality video lectures: high professional competence and good acting skills

“…It should be noted that, over the last 20 years, the number of publications dedicated to emotional aspects of learning is rather small as compared to cognitive or motivation aspects. Insufficient study of emotions in the research of multimodal learning is expressed in limited use of some data [22]. It also has been noted that objective data that may specifically concern emotional aspects of learning are most often ignored.…”