Investigating text‐picture integration in videos with the multimedia contradiction paradigm

Abstract

In two experiments, the multimedia contradiction paradigm was used to investigate whether learners map information conveyed through the audio and the picture track of a video. In Experiment 1 (N = 85), the information conveyed through the audio track and the picture track was always consistent (control group) or was made inconsistent by changing the audio track at one point in time (text‐wrong group). Experiment 2 (N = 143) added a second inconsistent condition by changing the picture track at one point in time (picture‐wrong group). In both experiments, the learners' gaze behaviour differed from that of the control group when inconsistent information was presented in the experimental groups. This indicates that mapping processes—which are an indispensable part of integration—occur when learners process videos. Regarding learning outcomes, no differences between groups were observed. In addition, only a few learners remembered the conflict after learning. Further, recall shifted towards the pictorial information when learners encountered conflict.

What makes video‐based academic lectures difficult for language learners to comprehend? The role of multimodal complexity

“…This finding is expected, as learners not only need to cope with the challenge of interpreting fast speech when dealing in a foreign language, but must also process and make sense of visual information. Previous studies have provided evidence that multimodal information processing is more challenging, particularly when the verbal and visual information in a video is inconsistent (Schüler & Merkt, 2021).…”