Event segmentation in continuous, naturalistic videos from model-based, data-driven, and human perspectives

Abstract

Human experience is characterised by an apparently continuous stream of information. However, when recalling our past, we commonly experience somewhat discrete episodes. Event segmentation has been suggested to provide a basis for episodic memory formation and recognition/recall. Previous work has suggested that event boundaries are reflected in shifts in stable patterns of neural activity across processing hierarchies, and that these boundaries can be detected using neuroimaging techniques. From a different direction, a recent series of studies in the domain of human time perception have shown that tracking event boundaries as "surprise" in network activity - in human perceptual processing or neural network proxies - successfully describes subjective reports of time on the scale of seconds to minutes. This project aims to understand how these methods of obtaining “event boundaries” perform relative to human-provided event annotations, in situations of life-like, naturalistic sensory stimulation. We compared the boundaries estimated from our neural-network-based model of time perception and a data-driven event segmentation approach (Greedy State Boundary Search on EEG data), against annotations from 131 human raters using two different inferential processes. In both methods, event boundaries from our model-based and the EEG data-driven approach performed similarly well, and were both indistinguishable from human provided boundaries in at least ~89% of cases examined. A random boundary generator was similar to human raters in only approximately 13% of cases. This work demonstrates the potential to reconcile the fields of time perception and episodic memory through a common foundation in the processes underlying event segmentation.

The Neural Basis of Event Segmentation: Stable Features in the Environment are Reflected by Neural States

“…It has previously been hypothesized that neural states should reflect meaningful units in the environment (Baldassano et al, 2017). However up to now, neural state boundaries have only been linked to event boundaries specifically (Baldassano et al, 2017; Geerligs et al, 2022; Mariola et al, 2022) and it has not been investigated if neural states at different levels of the temporal hierarchy indeed represent different meaningful units. In line with this hypothesis, we found that there is a significant alignment between changes in location and neural state boundaries in the place-sensitive PPA and RSC.…”

Towards an ecologically valid naturalistic cognitive neuroscience of episodic memory and event cognition: A literature review and synthesis

“…One could wonder if the neural patterns observed in the fMRI scanners in response to watching movies could be a faithful representation of the activations that occur over real-life event transitions. The practice of using movies for event segmentation studies then leads to a circular inference, illustrated by exploring the ability to segment scenes even though the events are already explicitly "given" to the participants (Mariola et al, 2022).…”

The influences of narrative perspective shift and scene detail on narrative semantic processing

“…An effective process of event segmentation is beneficial to individuals' understanding and memory of continuous information (Flores et al, 2017;Zacks et al, 2009). Event segmentation is based on top-down conceptual factors (e.g., comprehenders' goal, perspective, context knowledge or attentional focus) and bottom-up sensory factors (e.g., perceptual change of the object in narrative situation) (Bestgen & Vonk, 2000;Mariola et al, 2022;Zacks & Swallow, 2007). For example, comprehends will segment continuous narrative information in a storybook or film into discrete units based on their orientated attention to the perceptual change occurred in the aspects of character, space or time in the narrative (Bailey et al, 2017;Zacks et al, 2009).…”