Cross-codal integration of bridging-event information in narrative understanding

Huff, Markus; Rosenfelder, Dina; Oberbeck, Maria; Merkt, Martin; Papenmeier, Frank; Meitz, Tino

doi:10.3758/s13421-020-01039-z

Cited by 13 publications

(22 citation statements)

References 50 publications

(93 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This would be consistent with proposals that sustained negativities index processes of holding information in memory to resolve ambiguities for a mental model in discourse (Baggio 2018 ; van Berkum 2009 ). This ERP pattern is consistent with behavioral findings of slower self-paced viewing times for panels following an inference, including action stars, compared to those after explicit events (Cohn and Wittenberg 2015 ; Huff et al 2020 ; Hutson et al 2018 ; Magliano et al 2015 ). One such study introduced working memory load tasks between images prior to participants reaching the inference-generating panel (Magliano et al 2015 ), which interfered with bridging inference generation.…”

Section: Discussionsupporting

confidence: 85%

“…Such backward-looking processes have long been demonstrated in research on verbal discourse (McNamara and Magliano 2009), and comparable inferential processing has been posited across both verbal and visual narratives (Gernsbacher et al 1990;Loschky et al 2020;Magliano et al 2019). Indeed, as in studies of verbal discourse (McKoon and Ratcliff 1992), costs related to inferential processing manifest at the image following omitted-information as longer viewing times and increased visual search processes (Huff et al 2020;Hutson et al 2018;Magliano et al 2015Magliano et al , 2017. Such costs are modulated by interference to both linguistic and…”

Section: Introductionmentioning

confidence: 97%

See 1 more Smart Citation

A starring role for inference in the neurocognition of visual narratives

Cohn

2021

Cogn. Research

View full text Add to dashboard Cite

Research in verbal and visual narratives has often emphasized backward-looking inferences, where absent information is subsequently inferred. However, comics use conventions like star-shaped “action stars” where a reader knows events are undepicted at that moment, rather than omitted entirely. We contrasted the event-related brain potentials (ERPs) to visual narratives depicting an explicit event, an action star, or a “noise” panel of scrambled lines. Both action stars and noise panels evoked large N400s compared to explicit-events (300–500 ms), but action stars and noise panels then differed in their later effects (500–900 ms). Action stars elicited sustained negativities and P600s, which could indicate further interpretive processes and integration of meaning into a mental model, while noise panels evoked late frontal positivities possibly indexing that they were improbable narrative units. Nevertheless, panels following action stars and noise panels both evoked late sustained negativities, implying further inferential processing. Inference in visual narratives thus uses cascading mechanisms resembling those in language processing that differ based on the inferential techniques.

show abstract

Section: Discussionsupporting

confidence: 85%

Section: Introductionmentioning

confidence: 97%

A starring role for inference in the neurocognition of visual narratives

Cohn

2021

Cogn. Research

View full text Add to dashboard Cite

show abstract

“…For example, reading times for grammatically congruous substituted emoji were slower compared to words in sentences, but viewing times for grammatically incongruous or homophonous rebus emoji were even slower (Cohn et al, 2018;Scheffler et al, 2021). In addition, viewing times for sentences substituted for images in visual sequences (symmetrical substitution, as in Figure 9B), were also found to be slower than their substituted pictures (Huff et al, 2020). However, onomatopoeia in visual narratives were actually viewed faster than the pictures they substituted 1 .…”

Section: Substitutivementioning

confidence: 99%

“…An example of such Symmetrical Complex substitutive allocation would occur when a whole sentence replaces the Peak in a visual narrative sequence, as in Figure 9B . Like in the Asymmetrical example in Figure 9A , this Symmetrical Complex substitution replaces the Peak event in a visual narrative for text, only here the text is a whole sentence rather than a single unit (Cohn, 2019 ; Huff et al, 2020 ). This substitution has its own complex grammar (here simplified in notation) that uses categorical roles and constituent structure, but as a whole plays a categorical narrative role determined by the matrix grammar.…”

Section: Grammatical Relationshipsmentioning

confidence: 99%

Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture

Cohn

Schilperoord

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Language is typically embedded in multimodal communication, yet models of linguistic competence do not often incorporate this complexity. Meanwhile, speech, gesture, and/or pictures are each considered as indivisible components of multimodal messages. Here, we argue that multimodality should not be characterized by whole interacting behaviors, but by interactions of similar substructures which permeate across expressive behaviors. These structures comprise a unified architecture and align within Jackendoff's Parallel Architecture: a modality, meaning, and grammar. Because this tripartite architecture persists across modalities, interactions can manifest within each of these substructures. Interactions between modalities alone create correspondences in time (ex. speech with gesture) or space (ex. writing with pictures) of the sensory signals, while multimodal meaning-making balances how modalities carry “semantic weight” for the gist of the whole expression. Here we focus primarily on interactions between grammars, which contrast across two variables: symmetry, related to the complexity of the grammars, and allocation, related to the relative independence of interacting grammars. While independent allocations keep grammars separate, substitutive allocation inserts expressions from one grammar into those of another. We show that substitution operates in interactions between all three natural modalities (vocal, bodily, graphic), and also in unimodal contexts within and between languages, as in codeswitching. Altogether, we argue that unimodal and multimodal expressions arise as emergent interactive states from a unified cognitive architecture, heralding a reconsideration of the “language faculty” itself.

show abstract

“…Our study used a comic book in which text was extensively integrated with visual information. However, previous research (Huff et al, 2020) indicates that the combination of text and pictures in comics requires more effortful processing compared to purely pictorial Ó 2021 Hogrefe Publishing Journal of Media Psychology (2021)…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Narrative Media’s Emphasis on Distinct Moral Intuitions Alters Early Adolescents’ Judgments

Hahn

Tamborini

Prabhu

et al. 2022

Journal of Media Psychology

View full text Add to dashboard Cite

Abstract. Logic from the model of intuitive morality and exemplars (MIME) suggests that narrative media emphasizing moral intuitions can increase the salience of those intuitions in audiences. To date, support for this logic has been limited to adults. Across two studies, the present research tested MIME predictions in early adolescents (ages 10–14). The salience of care, fairness, loyalty, and authority intuitions was manipulated in a pilot study with verbal prompts ( N = 87) and in the main study with a comic book ( N = 107). In both studies, intuition salience was measured after induction. The pilot study demonstrated that exposure to verbal prompts emphasizing care, fairness, and loyalty increased the salience of their respective intuitions. The main study showed that exposure to comic books emphasizing all four separate intuitions increased salience of their respective intuitions in early adolescents. Results are discussed in terms of relevance for the MIME and understanding narrative media’s influence on children’s moral judgments.

show abstract

Cross-codal integration of bridging-event information in narrative understanding

Cited by 13 publications

References 50 publications

A starring role for inference in the neurocognition of visual narratives

A starring role for inference in the neurocognition of visual narratives

Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture

Narrative Media’s Emphasis on Distinct Moral Intuitions Alters Early Adolescents’ Judgments

Contact Info

Product

Resources

About