A starring role for inference in the neurocognition of visual narratives

Abstract

Research in verbal and visual narratives has often emphasized backward-looking inferences, where absent information is subsequently inferred. However, comics use conventions like star-shaped “action stars” where a reader knows events are undepicted at that moment, rather than omitted entirely. We contrasted the event-related brain potentials (ERPs) to visual narratives depicting an explicit event, an action star, or a “noise” panel of scrambled lines. Both action stars and noise panels evoked large N400s compared to explicit-events (300–500 ms), but action stars and noise panels then differed in their later effects (500–900 ms). Action stars elicited sustained negativities and P600s, which could indicate further interpretive processes and integration of meaning into a mental model, while noise panels evoked late frontal positivities possibly indexing that they were improbable narrative units. Nevertheless, panels following action stars and noise panels both evoked late sustained negativities, implying further inferential processing. Inference in visual narratives thus uses cascading mechanisms resembling those in language processing that differ based on the inferential techniques.

Picture perfect peaks: comprehension of inferential techniques in visual narratives

“…The consistently longer viewing times at the subsequent panel suggested that these viewing time differences did not pertain to deepness of processing, but rather to differences in surface-level structures. The faster viewing times made to action stars and onomatopoeias specifically seemed related to visual complexity of the panels themselves (Cohn, 2021; Cohn & Wittenberg, 2015). The results supported SPECT’s (Loschky et al, 2020) predictions that less content should require fewer eye fixations, and thus result in faster viewing times.…”

“…Among the inexplicit techniques, metaphors were also rated low, possibly due to the extended updating needed to interpret them, whereas action stars were rated high. Their ratings may have been motivated by their familiarity, as conventionalized units within comics (Cohn, 2021; Cohn & Wittenberg, 2015), arguably more so than metaphors. Moreover, action star sequences were viewed faster than metaphor sequences, and potentially comprehensibility ratings reflect the effort required for inference resolution.…”

“…[Framing] and [blend] both predict low scores, concurring with low ratings for onlookers and metaphors. Action stars are rated relatively high, most likely due to being a familiar part of the visual lexicon of comics (Cohn, 2021; Cohn & Wittenberg, 2015).…”

“…For all 30 strips, a cloze probability score (see Coderre et al, 2020) and inference assessment score (see Cohn, 2021) were measured in rating studies. For all participants, we measured their Visual Language Fluency Index (VLFI), a metric developed to assess comic reading expertise (Cohn, 2020a), with scores above 20 indicating a high comic fluency (i.e., the reader is well versed in reading comics), scores below 7 a low comic fluency, and an ideal average of around 12.…”

“…However, panels after action stars or blank panels both evoked equally longer viewing times, suggesting that the lack of explicit cues in both action stars and blank panels necessitates updating of the situation model. Recent work has further observed little difference in the brain responses between panels following action stars and ‘noise’ panels with scrambled lines (Cohn, 2021).…”