The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension

Abstract

Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500-700ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.

Reading Without Words: Eye Movements in the Comprehension of Comic Strips

“…may be expected in sequences that feature long-distance connections between panels, as in centre-embedded clauses (Cohn, 2013c), and in sequences that require "reanalysis" of prior information (Cohn, et al, 2014). Similar processes of integration have been shown with picture-text displays by Hegarty and Just (1993) and Holsanova et al, (2009).…”

“…Theoretical work has argued that coherent sequences are processed using a "narrative grammar" to order the meaning of sequential images. This grammar uses analogous functional principles as syntax uses to order the word units in sentences, such that sequential images take on categorical roles which are organised into constituent structures, only it does so at a discourse level of meaning (Cohn, Jackendoff, Holcomb, & Kuperberg, 2014). That is, though individual panel-units typically contain more information than individual wordunits-and thus are closer in meaning to whole sentences-similar "grammatical" principles operate to organize images and words into well-formed sequences (Cohn, 2013c;Cohn, et al, 2012).…”

“…That is, though individual panel-units typically contain more information than individual wordunits-and thus are closer in meaning to whole sentences-similar "grammatical" principles operate to organize images and words into well-formed sequences (Cohn, 2013c;Cohn, et al, 2012). This parallelism has been born out by experimentation measuring event-related brain potentials (ERPs), which has suggested that the same neural effects appear from the manipulation of this narrative grammar in sequential images as from manipulations of syntax in sentences (Cohn, et al, 2014;Cohn, et al, 2012). The processing of sequential images may therefore be similar to the processing of sequential words, even at a neurocognitive level, despite the differences in levels of…”

“…The stimuli consisted of panels culled from six volumes of the Complete Peanuts by Charles Schulz (1952)(1953)(1954)(1955)(1956)(1957)(1958)(1959)(1960)(1961)(1962)(1963)(1964)(1965)(1966)(1967)(1968), which were edited and recombined into novel sequences, and stimuli from this broader corpus have been used in several studies (Cohn, et al, 2014;Cohn & Paczynski, 2013;Cohn, et al, 2012). Novel sequences were generated to create sequences of six panels in length (as opposed to the standard four-panel daily strip length), and to reduce participants' familiarity with any specific sequences.…”

Framing Education: Doing Comics Literacy in the Classroom

“…Although the relations between visual literacy and comics literacy are never clearly defined in Hammond's work, she contributes by pointing out some literary functions of the comic book format, such as pacing, the possibility of stopping on certain panels and pages, or going back and rereading one or several panels, as well as the ability to overview several panels at once. Furthermore, some comics offer the possibility to disrupt the traditional reading order (left-to-right, or right-to-left, depending on your reading culture) through different panel structures (see also Cohn et al, 2014;Cohn et al, 2012). These structural options are not as easily applied to, for example, novels, so their use when working with comics requires further investigation.…”