A partially nested cortical hierarchy of neural states underlies event segmentation in the human brain

Abstract

A fundamental aspect of human experience is that it is segmented into discrete events. This may be underpinned by transitions between distinct neural states. Using an innovative data-driven state segmentation method, we investigate how neural states are organized across the cortical hierarchy and where in cortex neural state and perceived event boundaries overlap. Our results show that neural state boundaries are organized in a temporal cortical hierarchy, with short states in primary sensory regions and long states in anterior temporal pole and lateral and medial prefrontal cortex. Neural state boundaries overlap with event boundaries across large parts of this hierarchy. State boundaries are shared within and between groups of brain regions that resemble well known functional networks, such as the default mode network that fractionates into two subnetworks; one fast, one slow. Together these findings suggest that a nested cortical hierarchy of neural states forms the basis of event segmentation.

A generalized cortical activity pattern at internally-generated mental context boundaries during unguided narrative recall

“…Relatedly, a recent human neurophysiological study (Zheng et al, 2022) reported that medial temporal cortex neurons distinguished within- and between-movie boundaries while subjects were watching short video clips; some neurons responded only to between-movie boundaries, whereas a separate group of neurons responded to both types of boundaries. These findings may be in line with the view that event boundaries have a hierarchical structure, with different brain areas along the information pathway reflecting different levels of boundaries, from fine-grained sensory transitions to coarse-grained situational transitions (Baldassano et al, 2017; Chang et al, 2021; Geerligs et al, 2021). However, it is still puzzling that within- and between-movie boundaries in our study produced qualitatively distinct neural patterns within a highest-order area (PMC), even though both categories consisted of prominent boundaries between situations spanning tens of seconds to several minutes.…”

“…Considering these findings and the widely-known involvement of the DMN in internally-oriented cognition (e.g., Addis et al, 2007;Andrews-Hanna et al, 2010;Christoff et al, 2009) together, it has been suggested that the DMN integrates both internal and external information to represent and maintain an abstract mental model of the current situation or state (Stawarczyk et al, 2021;Yeshurun et al, 2021); located furthest away from sensory-motor areas (Smallwood et al, 2021), the DMN integrates information across different modalities (Bonnici et al, 2016;Ramanan et al, 2018) and over long timescales (Chang et al, 2021;Hasson et al, 2015). Supporting this idea, neural activation patterns in sub-regions of the DMN, especially PMC, tend to persist for extended periods of time during naturalistic movie watching, and transitions between these persistent neural states coincide with perceived event boundaries (Baldassano et al, 2017;Geerligs et al, 2021). Our study extends this finding by identifying a transient, boundary-induced phenomenon which is a unique and independent state represented in the DMN.…”

Information flow across the cortical timescales hierarchy during narrative construction

“…The chunking of information at varying granularity is supported by recent studies that used data-driven methods to detect boundaries as shifts between stable patterns of brain activity. These studies revealed a nested hierarchy of cortical events, from brief events in sensory regions to longer-duration events in high-order areas comprising the DMN (Baldassano et al, 2017;Geerligs et al, 2021).…”

A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall

“…Considering these findings and the widely known involvement of the DMN in internally oriented cognition (e.g., Addis et al, 2007 ; Andrews-Hanna et al, 2010 ; Christoff et al, 2009 ) together, it has been suggested that the DMN integrates both internal and external information to represent and maintain an abstract mental model of the current situation or state ( Stawarczyk et al, 2021 ; Yeshurun et al, 2021 ); located furthest away from sensorimotor areas ( Smallwood et al, 2021 ), the DMN integrates information across different modalities ( Bonnici et al, 2016 ; Ramanan et al, 2018 ) and over long timescales ( Chang et al, 2021 ; Hasson et al, 2015 ). Supporting this idea, neural activation patterns in subregions of the DMN, especially PMC, tend to persist for extended periods of time during naturalistic movie watching, and transitions between these persistent neural states coincide with perceived event boundaries ( Baldassano et al, 2017 ; Geerligs et al, 2021 ). Our study extends this finding by identifying a transient, boundary-induced phenomenon, which is a unique and independent state represented in the DMN.…”