What's your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing

Abstract

Visual narratives sometimes depict successive images with different characters in the same physical space; corpus analysis has revealed that this occurs more often in Japanese manga than American comics. We used event-related brain potentials to determine whether comprehension of “visual narrative conjunctions” invokes not only incremental mental updating as traditionally assumed, but also, as we propose, “grammatical” combinatoric processing. We thus crossed (non)/conjunction sequences with character (in)/congruity. Conjunctions elicited a larger anterior negativity (300–500 ms) than nonconjunctions, regardless of congruity, implicating “grammatical” processes. Conjunction and incongruity both elicited larger P600s (500–700 ms), indexing updating. Both conjunction effects were modulated by participants’ frequency of reading manga while growing up. Greater anterior negativity in frequent manga readers suggests more reliance on combinatoric processing; larger P600 effects in infrequent manga readers suggest more resources devoted to mental updating. As in language comprehension, it seems that processing conjunctions in visual narratives is not just mental updating but also partly grammatical, conditioned by comic readers’ experience with specific visual narrative structures.Electronic supplementary materialThe online version of this article (doi:10.1186/s41235-017-0064-5) contains supplementary material, which is available to authorized users.

Visual narrative comprehension: Universal or not?

“…Behavioral methods suggest that language and visual narratives share resources for inference generation (Magliano, Larson, Higgs, & Loschky, 2015) and segmentation (Magliano, Kopp, McNerney, Radvansky, & Zacks, 2012). Neurocognitive research has also implicated similar brain areas across verbal and visual narratives (Gernsbacher & Robertson, 2004;Robertson, 2000), including Broca's and Wernicke's areas (Cohn & Maher, 2015;Nagai, Endo, & Takatsune, 2007;Osaka, Yaoi, Minamoto, & Osaka, 2014).…”

In defense of a “grammar” in the visual language of comics

“…Similar brainwaves have been observed in sequential images: larger N400s appear to incongruous than congruous images within a narrative sequence (West and Holcomb, 2002) or to images within scrambled than coherent sequences (Cohn et al, 2012). In contrast, different brainwaves responses associated with the processing of syntax are evoked by violations of narrative grammar, such as the omission of a particular narrative category (Cohn and Kutas, 2015), the disruption of narrative constituents (Cohn et al, 2014), or the contrast between narrative patterns (Cohn and Kutas, 2017). Thus, different brain responses characteristic of grammar and meaning are evoked by different types of violations, as predicted by VNG.…”

“…VNG agrees with the idea that meaningful relations between units must be updated as one progresses through a sequence, as would be consistent with psychological theories of discourse (Magliano and Zacks, 2011;Zwaan and Radvansky, 1998). Empirical work has shown that discontinuity in meaning triggers a brainwave response argued to be associated with mental model updating (Donchin and Coles, 1988;Kuperberg, 2013), which in sequential images is modulated by changes in characters (Cohn and Kutas, 2017), the generation of inference Kutas, 2015, 2017), and incongruity to the structure of actions (Amoruso et al, 2013;Sitnikova et al, 2008), such as images with omitted or incongruous motion lines (Cohn and Maher, 2015). This updating process appears continuous and ongoing throughout a visual narrative (Cohn and Kutas, 2015;Osaka et al, 2014).…”

“…Recent research has also suggested that the processing of visual narratives may vary based on the patterns found in the particular "visual language" that a person reads. In a recent study, participants' brainwaves were recorded to panels in sequences that did or did not use E-Conjunction (Cohn and Kutas, 2017). Participants evoked two neural responses.…”

“…2a), and thus we must infer this shared spatial environment. Such a process of inference occurs by updating a mental model of the scene, which interfaces with the narrative pattern (Cohn and Kutas, 2017). This type construction is called Environmental-Conjunction or E-Conjunction since the conjunction (repetition of panels playing the same narrative role) maps to an inference of a broader spatial environment (notated with subscript "e").…”