Information flow across the cortical timescales hierarchy during narrative construction

Abstract

When listening to spoken narratives, we must integrate information over multiple, concurrent timescales, building up from words to phrases to sentences to a coherent narrative. Recent evidence suggests that the brain relies on a chain of hierarchically organized areas with increasing temporal receptive windows to process naturalistic narratives. In this study, we use inter-subject functional connectivity to reveal a stimulus-driven information flow along the cortical hierarchy. Using cross-correlation analysis to estimate the time lags between six functional networks, we found a fixed temporal sequence of information flow, starting in early auditory areas, followed language areas, the attention network, and lastly the default mode network. This gradient is consistent across eight distinct stories but absent in resting-state and scrambled story data, indicating that the lag gradient reflects the construction of narrative features. Finally, we simulate a variety of narrative integration models and demonstrate that nested narrative structure along with the gradual accumulation of information within the boundaries of linguistic events at each level of the processing hierarchy is sufficient to reproduce the lag gradient. Taken together, this study provides a computational framework for how information flows along the cortical hierarchy during narrative comprehension.

A generalized cortical activity pattern at internally-generated mental context boundaries during unguided narrative recall

“…Relatedly, a recent human neurophysiological study (Zheng et al, 2022) reported that medial temporal cortex neurons distinguished within- and between-movie boundaries while subjects were watching short video clips; some neurons responded only to between-movie boundaries, whereas a separate group of neurons responded to both types of boundaries. These findings may be in line with the view that event boundaries have a hierarchical structure, with different brain areas along the information pathway reflecting different levels of boundaries, from fine-grained sensory transitions to coarse-grained situational transitions (Baldassano et al, 2017; Chang et al, 2021; Geerligs et al, 2021). However, it is still puzzling that within- and between-movie boundaries in our study produced qualitatively distinct neural patterns within a highest-order area (PMC), even though both categories consisted of prominent boundaries between situations spanning tens of seconds to several minutes.…”

“…Prior studies have shown that the DMN responds to external context transitions including experimental task switching (Crittenden et al, 2015; Smith et al, 2018) as well as event boundaries in movie clips (Reagh et al, 2020; Speer et al, 2007). Considering these findings and the widely-known involvement of the DMN in internally-oriented cognition (e.g., Addis et al, 2007; Andrews-Hanna et al, 2010; Christoff et al, 2009) together, it has been suggested that the DMN integrates both internal and external information to represent and maintain an abstract mental model of the current situation or state (Stawarczyk et al, 2021; Yeshurun et al, 2021); located furthest away from sensory-motor areas (Smallwood et al, 2021), the DMN integrates information across different modalities (Bonnici et al, 2016; Ramanan et al, 2018) and over long timescales (Chang et al, 2021; Hasson et al, 2015). Supporting this idea, neural activation patterns in sub-regions of the DMN, especially PMC, tend to persist for extended periods of time during naturalistic movie watching, and transitions between these persistent neural states coincide with perceived event boundaries (Baldassano et al, 2017; Geerligs et al, 2021).…”

A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall

“…Relatedly, a recent human neurophysiological study ( Zheng et al, 2022 ) reported that medial temporal cortex neurons distinguished within- and between-movie boundaries while subjects were watching short video clips; some neurons responded only to between-movie boundaries, whereas a separate group of neurons responded to both types of boundaries. These findings may be in line with the view that event boundaries have a hierarchical structure, with different brain areas along the information pathway reflecting different levels of boundaries, from fine-grained sensory transitions to coarse-grained situational transitions ( Baldassano et al, 2017 ; Chang et al, 2021 ; Geerligs et al, 2021 ). However, it is still puzzling that within- and between-movie boundaries in our study produced qualitatively distinct neural patterns within a highest-order area (PMC), even though both categories consisted of prominent boundaries between situations spanning tens of seconds to several minutes.…”

“…Prior studies have shown that the DMN responds to external context transitions, including experimental task switching ( Crittenden et al, 2015 ; Smith et al, 2018 ) as well as event boundaries in movie clips ( Reagh et al, 2020 ; Speer et al, 2007 ). Considering these findings and the widely known involvement of the DMN in internally oriented cognition (e.g., Addis et al, 2007 ; Andrews-Hanna et al, 2010 ; Christoff et al, 2009 ) together, it has been suggested that the DMN integrates both internal and external information to represent and maintain an abstract mental model of the current situation or state ( Stawarczyk et al, 2021 ; Yeshurun et al, 2021 ); located furthest away from sensorimotor areas ( Smallwood et al, 2021 ), the DMN integrates information across different modalities ( Bonnici et al, 2016 ; Ramanan et al, 2018 ) and over long timescales ( Chang et al, 2021 ; Hasson et al, 2015 ). Supporting this idea, neural activation patterns in subregions of the DMN, especially PMC, tend to persist for extended periods of time during naturalistic movie watching, and transitions between these persistent neural states coincide with perceived event boundaries ( Baldassano et al, 2017 ; Geerligs et al, 2021 ).…”

Effects of Connectivity on Narrative Temporal Processing in Structured Reservoir Computing

“…The current research addresses a dimension of language processing that has gained increasing visibilitythe temporal processing of language in a cortical hierarchy [4][5][6][7][8]. Intuitively, sensory-driven cortical responses should be rapid and reflect the temporal structure of the input, while cortical activity underlying higher level cognitive functions that may involve reasoning and memory will have longer time-scales of processing [9].…”