Various studies have demonstrated an advantage of auditory over visual text modality when learning with texts and pictures. To explain this modality effect, two complementary assumptions are proposed by cognitive theories of multimedia learning: first, the visuospatial load hypothesis, which explains the modality effect in terms of visuospatial working memory overload in the visual text condition; and second, the temporal contiguity assumption, according to which the modality effect occurs because solely auditory texts and pictures can be attended to simultaneously. The latter explanation applies only to simultaneous presentation, the former to both simultaneous and sequential presentation. This paper introduces a third explanation, according to which parts of the modality effect are due to early, sensory processes. This account predicts that-for texts longer than one sentence-the modality effect with sequential presentation is restricted to the information presented most recently. Two multimedia experiments tested the influence of text modality across three different conditions: simultaneous presentation of texts and pictures versus sequential presentation versus presentation of text only. Text comprehension and picture recognition served as dependent variables. An advantage for auditory texts was restricted to the most recent text information and occurred under all presentation conditions. With picture recognition, the modality effect was restricted to the simultaneous condition. These findings clearly support the idea that the modality effect can be attributed to early processes in perception and sensory memory rather than to a working memory bottleneck.
Auditory text presentation improves learning with pictures and texts. With sequential text-picture presentation, cognitive models of multimedia learning explain this modality effect in terms of greater visuo-spatial working memory load with visual as compared to auditory texts. Visual texts are assumed to demand the same working memory subsystem as pictures, while auditory texts make use of an additional cognitive resource. We provide two alternative assumptions that relate to more basic processes: First, acoustic-sensory information causes a retention advantage for auditory over visual texts which occurs no matter if a picture is presented or not. Second, eye movements during reading hamper visuo-spatial rehearsal. Two experiments applying elementary procedures provide first evidence for these assumptions. Experiment 1 demonstrates that, regarding text recall, the auditory advantage is independent of visuo-spatial working memory load. Experiment 2 reveals worse matrix recognition performance after reading text requiring eye movements than after listening or reading without eye movements.A crucial finding in cognitive multimedia research is the so-termed modality effect according to which learning and memory performance are better when pictures are accompanied by auditory rather than by visual texts (e.g. Mayer & Moreno, 1998;Moreno & Mayer, 1999;Mousavi, Low, & Sweller, 1995; for meta-analyses cf. Ginns, 2005). This effect is attributed to two different aspects of the learning process which are referred to by the most influential cognitive theory concerned with multimedia learning (e.g. Mayer, 2001). The split attention assumption concerns restrictions of the attentional system: Auditory texts and pictures can be focused on (and processed) at the same time, while visual texts and pictures have to be focused on (and processed) sequentially. The second explanation refers to modality-specific demands on working memory, and further on, we will refer to it as visuo-spatial working memory load hypothesis, or shorter, visuo-spatial load hypothesis. According to this assumption, visual texts and pictures have to be represented (and processed) in the same working memory component, the visuo-spatial system. In contrast, auditory texts are represented (and processed) in an additional APPLIED COGNITIVE PSYCHOLOGY
Abstract.Two experiments were conducted to replicate the modality effect and to test two opposing theoretical explanations: From the visuo-spatial load hypothesis (as implied by the Cognitive Theory of Multimedia Learning), we derived the prediction that the modality effect depends on the need to simultaneously store and process text and pictorial information in the same working memory subsystem, which implies that the modality effect is not influenced by text length. However, based on the concurrent auditory recency hypothesis, we expected the modality effect to occur with very short texts only, irrespective of whether a picture is present. Adopting a German translation of the learning materials used by Moreno and Mayer (1999) , we failed to detect a modality effect for sequential text-picture-presentation in Experiment 1. In Experiment 2, a modality effect was expected for simultaneous presentation of the same texts and pictures, yet again remained absent. We conclude with the suggestion that the apparent robustness of the modality effect reported in the literature might be explained – at least in part – by publication bias.
This work contributes to the understanding of the visual similarity effect in verbal working memory, a finding that suggests that the visuo-spatial sketch pad-the system in Baddeley's working memory model specialised in retaining nonverbal visual information-might be involved in the retention of visually presented verbal materials. Crucially this effect is implicitly interpreted by the most influential theory of multimedia learning as evidence for an obligatory involvement of the visuo-spatial sketch pad. We claim that it is only involved when the functioning of the working memory component normally used for processing verbal material is impaired. In this article we review the studies that give rise to the idea of obligatory involvement of the visuo-spatial sketch pad and suggest that some findings can be understood with reference to orthographic rather than visual similarity. We then test an alternative explanation of the finding that is most apt to serve as evidence for obligatory involvement of the visuo-spatial sketch pad. We conclude that, in healthy adults and under normal learning conditions, the visual similarity effect can be explained within the framework of verbal working memory proposed by Baddeley (e.g., 1986, 2000) without additional premises regarding the visuo-spatial sketch.
Liegt der Modalitätseffekt beim Lernen mit Texten und Bildern darin begründet, dass Bild und visueller Text gleichzeitig im visuell-räumlichen Arbeitsgedächtnissystem verarbeitet werden und dieses dadurch überfordert ist? Das in diesem Artikel vorgestellte Experimentalparadigma erlaubt die Überprüfung einer alternativen Erklärung, die sich aus der Grundlagenforschung zum verbalen Behalten ergibt. Als Material dienten Items, die aus jeweils einem kurzen Text zu einem anschließend präsentierten Pseudo-Sternbild bestanden. Als kritischer Faktor wurde die Textmodalität (auditiv vs. visuell) zwischen Versuchspersonen variiert. Die Probanden beantworteten zu jedem Text vier Fragen und reproduzierten dann das Sternbild. Der einzig signifikante Einfluss der Textmodalität fand sich in Form einer Interaktion mit der Position der Textinformation: Nur die zuletzt präsentierte Information wurde in der auditiven Bedingung besser erinnert als in der visuellen. Dieser Befund steht in Einklang mit der Annahme eines behaltenswirksamen akustisch-sensorischen Codes, der nur bei auditiver Präsentation verfügbar ist, und auch dort nur für Information, die noch nicht überschrieben wurde – allerdings unabhängig von der visuell-räumlichen Arbeitsgedächtnisbelastung.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.