The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives

Abstract

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world.

The influence of sequential predictions on scene-gist recognition

“…Facilitation was also found for targets with visually similar primes. These results are consistent with hypotheses generated from SPECT that the extent to which facilitation will be found depends upon the degree of spatiotemporal coherence between the contents of the current event model and new scene information (Loschky et al, 2019).…”

“…According to SPECT, one's understanding in the back end of what is happening in the current event (e.g., navigating through one's office toward the doorway) influences information extraction in the front end (Loschky et al, 2018;Loschky et al, 2019). In Experiment 1, we found that scenes presented in coherent sequences, which enabled observers to create a spatiotemporally coherent event model, were identified more accurately than scenes presented in randomized sequences.…”

“…The first was to determine the extent to which the scene categories in coherent and randomized sequences were predictable. One possibility, according to SPECT, is that the presentation of the first scene in a coherent image sequence lays the foundation for the current event model in working memory (Loschky et al, 2019). Once an event model is constructed, upcoming scene categories become predictable.…”

“…Our interest in this problem comes from the recently developed theoretical framework of the Scene Perception & Event Comprehension Theory (SPECT; Loschky, Hutson, Smith, Smith, & Magliano, 2018;Loschky, Larson, Smith, & Magliano, 2019;Magliano, Loschky, Clinton, & Larson, 2013). SPECT considers scene-gist recognition as one part of the larger problem of perceiving and understanding scenes and events in the real world.…”

“…Scene gist plays an important role in both front-and backend processes. The gist of a scene is recognized within a single eye fixation during the process of information extraction in the front end (Greene & Oliva, 2009;Larson, 2012), which is important for laying the foundation of the current event model in the back end (Loschky et al, 2019). Importantly, SPECT proposes that back-end processes involved in event-model construction influence front-end processes involved in information extraction.…”