Reading Without Words: Eye Movements in the Comprehension of Comic Strips

Abstract

The study of attention in pictures is mostly limited to individual images. When we "read" a visual narrative (e.g., a comic strip), the pictures have a coherent sequence, but it is not known how this affects attention. In two experiments, we eyetracked participants in order to investigate how disrupting the visual sequence of a comic strip would affect attention.Both when panels were presented one at a time (Experiment 1) and when a sequence was presented all together (Experiment 2), pictures were understood more quickly and with fewer fixations when in their original order. When order was randomised, the same pictures required more attention and additional "regressions". Fixation distributions also differed when the narrative was intact, showing that context affects where we look. This reveals the role of top-down structures when we attend to pictorial information, as well as providing a springboard for applied research into attention within image sequences.3"

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

“…This decoding involves attentional selection to guide object and scene perception to extract the relevant cues of an image (Loschky et al., , ). In many cases, properties of the images themselves motivate such cues, as readers by and large focus on the same visual regions of interest whether images belong to a coherent or scrambled sequence (Foulsham, Wybrow, & Cohn, ).…”

“…Nevertheless, most images in visual narratives are created (i.e., drawn ) intentionally to belong to a sequence, and readers in turn are tasked with finding the specific cues relevant for that context. In doing so, readers seem fairly directed in filtering the relevant content of a panel's sequential understanding (Foulsham et al., ; Laubrock, Hohenstein, & Kümmerer, ), and comprehension of sequential images persists even at fairly rapid exposure (Hagmann & Cohn, ; Inui & Miyamoto, ). This means that the extraction of relevant cues must happen quickly and with insight from context (below).…”

“…As a reader progresses panel by panel in a coherent narrative sequence, semantic access of each image becomes facilitated as structural and semantic expectancies are cyclically confirmed relative to what came before (Cohn et al., ; Giglio, Minati, & Boggio, ). This buildup is evident in shorter viewing times and attenuated N400s across each subsequent image in a coherent sequence (Cohn & Paczynski, ; Cohn & Wittenberg, ; Cohn et al., ; Foulsham et al., ; Giglio et al., ). Thus, exposure to a prior congruous context makes subsequent information easier to process.…”

The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives

“…A key difference is that models are not given text from the final panel; in text cloze, models are allowed to look at the final panel's artwork. This design is motivated by eyetracking studies in single-panel cartoons, which show that readers look at artwork before reading the text [7], although atypical font style and text length can invert this order [16].…”

In defense of a “grammar” in the visual language of comics

“…It is worth noting that B&W's example (Fig. 8e) scrambles the panels of a sequence to appear out of order; scrambling all or some panels is one of the most basic manipulations used in experiments on sequential images, and has consistently yielded both conscious and unconscious responses of incoherence (Cohn and Wittenberg, 2015;Foulsham et al, 2016;Gernsbacher et al, 1990;Hagmann and Cohn, 2016;Inui and Miyamoto, 1981;Nagai et al, 2007;Osaka et al, 2014).…”