Tea With Milk? A Hierarchical Generative Framework of Sequential Event Comprehension

Abstract

To make sense of the world around us, we must be able to segment a continual stream of sensory inputs into discrete events. In this review, I propose that in order to comprehend events, we engage hierarchical generative models that “reverse engineer” the intentions of other agents as they produce sequential action in real time. By generating probabilistic predictions for upcoming events, generative models ensure that we are able to keep up with the rapid pace at which perceptual inputs unfold. By tracking our certainty about other agents' goals and the magnitude of prediction errors at multiple temporal scales, generative models enable us to detect event boundaries by inferring when a goal has changed. Moreover, by adapting flexibly to the broader dynamics of the environment and our own comprehension goals, generative models allow us to optimally allocate limited resources. Finally, I argue that we use generative models not only to comprehend events but also to produce events (carry out goal‐relevant sequential action) and to continually learn about new events from our surroundings. Taken together, this hierarchical generative framework provides new insights into how the human brain processes events so effortlessly while highlighting the fundamental links between event comprehension, production, and learning.

Emergent Goal-Anticipatory Gaze in Infants via Event-Predictive Learning and Inference

“…Between these boundaries, the hand typically follows a straight trajectory towards the object. Evidence for such event representations stems from different disciplines and can be found on different levels of processing, ranging from sensorimotor activations to semantic and linguistic representations (Baldwin & Kosie, 2020;Butz et al, 2020;Cooper, 2019;Franklin, Norman, Ranganath, Zacks, & Gershman, 2020;Kuperberg, 2020).…”

Predictive coding across the left fronto-temporal hierarchy during language comprehension

“…This higher-level prediction error initiated the retrieval of a new schema from long-term memory 18 , enabling comprephenders to successfully shift their event model, and resolve the error 22,39 . The updated event model, in turn, provided retroactive feedback to the left temporal cortex, enhancing activity over schema-consistent lexico-semantic representations, while reducing activity over incorrectly predicted lexico-semantic information 15 . The top-down nature of this feedback enhancement may explain why, within this late time window, the dipoles within the temporal cortex were of the opposite polarity to those produced by the bottom-up prediction error within the 300-500ms time window.…”

“…Importantly, however, if an update leads the comprehender to infer an event that is either inconsistent with a prior high-certainty event representation, or that falls outside the range of plausible events reconstructed by the higher-level schema, then it will produce a higher-level event prediction error. This event prediction error induces a shift away from the current schema at the highest level of the hierarchy 15 . If there is a new schema stored within long-term memory that can better explain the input, then it will be retrieved 18 , resulting in the production of new reconstructions that provide feedback to lower cortical levels, enhancing activity over schema-consistent lexico-semantic representations 15 .…”

Dynamic predictive coding across the left fronto-temporal language hierarchy: Evidence from MEG, EEG and fMRI

“…Importantly, however, if an update leads the comprehender to infer an event that is either inconsistent with a prior high-certainty event representation, or that falls outside the range of plausible events reconstructed by the higher-level schema, then it will produce a higher-level event prediction error. This event prediction error induces a shift away from the current schema at the highest level of the hierarchy 15 . If there is a new schema stored within long-term memory that can better explain the input, then it will be retrieved 18 , resulting in the production of new reconstructions that provide feedback to lower cortical levels, enhancing activity over schema-consistent lexico-semantic representations 15 .…”

“…This event prediction error induces a shift away from the current schema at the highest level of the hierarchy 15 . If there is a new schema stored within long-term memory that can better explain the input, then it will be retrieved 18 , resulting in the production of new reconstructions that provide feedback to lower cortical levels, enhancing activity over schema-consistent lexico-semantic representations 15 . If, however, the newly inferred event is completely anomalous, with no pre-stored schema that can explain it, then this will result in a failure to switch off prediction error at still lower levels of the hierarchy (reanalysis), and may trigger new learning in order to explain the input 18,19 .…”