The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension

Abstract

This study investigated the relative roles of visuospatial versus linguistic working memory (WM) systems in the online generation of bridging inferences while viewers comprehend visual narratives. We contrasted these relative roles in the visuospatial primacy hypothesis versus the shared (visuospatial & linguistic) systems hypothesis, and tested them in 3 experiments. Participants viewed picture stories containing multiple target episodes consisting of a beginning state, a bridging event, and an end state, respectively, and the presence of the bridging event was manipulated. When absent, viewers had to infer the bridging-event action to comprehend the end-state image. A pilot study showed that after viewing the end-state image, participants' think-aloud protocols contained more inferred actions when the bridging event was absent than when it was present. Likewise, Experiment 1 found longer viewing times for the end-state image when the bridging-event image was absent, consistent with viewing times revealing online inference generation processes. Experiment 2 showed that both linguistic and visuospatial WM loads attenuated the inference viewing time effect, consistent with the shared systems hypothesis. Importantly, however, Experiment 3 found that articulatory suppression did not attenuate the inference viewing time effect, indicating that (sub)vocalization did not support online inference generation during visual narrative comprehension. Thus, the results support a shared-systems hypothesis in which both visuospatial and linguistic WM systems support inference generation in visual narratives, with the linguistic WM system operating at a deeper level than (sub)vocalization.

Understanding Moment‐to‐Moment Processing of Visual Narratives

“…The viewing time cleaning followed the same procedure as Magliano, Larson, et al. (). This criterion‐based cleaning set the minimum viewing time to 480 ms.…”

“…. While there is some debate as to what aspects of comprehension are modality independent (e.g., Loughlin, Grossnickle, Dinsmore, & Alexander, ), there is little doubt that bridging inferences are important for comprehension regardless of the modality of a narrative (Magliano, Higgs, & Clinton, in press; Magliano, Loschky, Clinton, & Larson, ; Magliano, Larson, et al., ).…”

“…In fact, recent EEG work with sequential narratives has shown effects consistent with inference generation results in both updating the narrative mental model and activating information in long‐term memory (Cohn & Kutas, ). However, it is also possible that processes that support the understanding of complex visual scenes, and specifically those that support scene perception (Henderson & Hollingworth, ), are also enlisted to support bridging inferences in graphic narratives (Magliano, Larson, et al., ). This study explores this possibility.…”

The role of working memory in inference generation during reading comprehension: Retention, (re)activation, or suppression of verbal information?

“…Notwithstanding, the correlations observed with reading or listening-span measures had a general advantage over those observed with WM span tests that required the processing of single verbal items outside of a discourse context (e.g., Daneman & Merikle, 1996;Leather & Henry, 1994;Seigneuric et al, 2000). Two other studies, which applied a concurrent WM load task to explore the role of WM in narrative comprehension, found support for the involvement of visuospatial (Fincher-Kiefer & D'Agostino, 2004;Magliano, Larson, Higgs, & Loschky, 2016) and verbal WM resources (Magliano et al, 2016) in inference generation. In contrast to the present findings, the activation of predictive (but not bridging) inferences was impaired when participants had to retain visual dots' locations in WM while they read short narratives (Fincher-Kiefer & D'Agostino, 2004), and the activation of bridging inferences was impaired when participants had to retain visual dots' locations or lists of words in WM while they viewed picture stories (Magliano et al, 2016).…”

“…Two other studies, which applied a concurrent WM load task to explore the role of WM in narrative comprehension, found support for the involvement of visuospatial (Fincher-Kiefer & D'Agostino, 2004;Magliano, Larson, Higgs, & Loschky, 2016) and verbal WM resources (Magliano et al, 2016) in inference generation. In contrast to the present findings, the activation of predictive (but not bridging) inferences was impaired when participants had to retain visual dots' locations in WM while they read short narratives (Fincher-Kiefer & D'Agostino, 2004), and the activation of bridging inferences was impaired when participants had to retain visual dots' locations or lists of words in WM while they viewed picture stories (Magliano et al, 2016). Apparently, more research is needed to explore the role of visuospatial WM resources in predictive versus bridging inferencing using individual differences versus concurrent task methods.…”