Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems

Baram, Alon B.; Muller, Timothy H.; Nili, Hamed; Garvert, Mona M.; Behrens, Timothy E.J.

doi:10.1016/j.neuron.2020.11.024

Cited by 89 publications

(88 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This indicates that sequential reactivation is able to combine prior learning with new sensory inputs to produce behavior relevant to new environments. This aligns with the theoretical proposal that decomposing information into separate structural and sensory representations can be used to generalize structural relationships to situations with different sensory specifics but similar underlying structure (Behrens et al, 2018;Baram et al, 2020;Whittington et al, 2020). Nonetheless, this theoretical view is not restricted to replay events and, as a result, there is still limited work testing this idea in the context of replay.…”

Section: Inference and Generalizationsupporting

confidence: 80%

“…The clustering of trajectories seen in the model above is related to a broader theoretical view, which has emphasized that separate encoding of transition information and sensory information during learning will allow knowledge about transitions to be reused across situations with structural similarities but new sensory specifics (Behrens et al, 2018;Baram et al, 2020;Whittington et al, 2020). Because replay provides a strong candidate mechanism for learning about transition structure (Stoianov et al, 2020), replay of abstract (sensory-independent) transition information could help to build representations of task structure that can be generalized and used to guide behaviour in new sensory environments or combined with sensory observations to make inferences about the current environment (Evans & Burgess, 2019;Stoianov et al, 2020).…”

Section: Inference and Generalizationmentioning

confidence: 94%

See 1 more Smart Citation

Replay in minds and machines

Wittkuhn¹,

Chien²,

Hall-McMaster³

2021

Preprint

View full text Add to dashboard Cite

Experience-related brain activity patterns have been found to reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research have significantly expanded our understanding of the potential computational benefits replay may provide to biological and artificial agents alike. We provide an overview of findings in replay research from neuroscience and machine learning and summarize the computational benefits an agent can gain from replay that cannot be achieved through direct interactions with the world itself. These benefits include faster learning and data efficiency, less forgetting, prioritizing important experiences, as well as improved planning and generalization. In addition to the benefits of replay for improving an agent's decision-making policy, we highlight the less-well studied aspect of replay in representation learning, wherein replay could provide a mechanism to learn the structure and relevant aspects of the environment. Thus, replay might help the agent to build task-appropriate state representations.

show abstract

Section: Inference and Generalizationsupporting

confidence: 80%

Section: Inference and Generalizationmentioning

confidence: 94%

Replay in minds and machines

Wittkuhn¹,

Chien²,

Hall-McMaster³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These ideas were developed in the context of the entorhinal cortex. Whilst we did not record from entorhinal cortex in the current study, recent fMRI evidence in humans in a conceptually similar experiment suggests entorhinal representations will also generalise the structure of reinforcement learning tasks 10 . It is also notable that abstract representations of trials are present in mPFC in purely spatial contexts 35,36 .…”

Section: Discussionmentioning

confidence: 91%

“…However, recent data suggest that interactions between frontal cortex and the hippocampal formation play an important role. Neurons 7,8 and fMRI voxels 9,10 in these brain regions form representations that generalise over different sensorimotor examples of tasks with the same structure, and track different task rules embedded in otherwise similar sensory experience 11 .…”

Section: Introductionmentioning

confidence: 99%

Complementary task representations in hippocampus and prefrontal cortex for generalising the structure of problems

Samborska

et al. 2021

Preprint

View full text Add to dashboard Cite

Few situations in life are completely novel. We effortlessly generalise prior knowledge to solve novel problems, abstracting common structure and mapping it onto new sensorimotor specifics. Here we trained mice on a series of reversal learning tasks that shared the same structure but had different physical implementations. Performance improved across tasks, demonstrating transfer of knowledge. Neurons in medial prefrontal cortex (mPFC) maintained similar representations across multiple tasks, despite their different sensorimotor correlates, whereas hippocampal (dCA1) representations were more strongly influenced by the specifics of each task. Critically, this was true both for representations of the events that comprised each trial, and those that integrated choices and outcomes over multiple trials to guide subjects' decisions. These data suggest that PFC and hippocampus play complementary roles in generalisation of knowledge, with the former abstracting the common structure among related tasks, and the latter mapping this structure onto the specifics of the current situation.

show abstract

“…Finally, recent research focused on how humans extract abstract knowledge, and generalize this knowledge to other situations (Collins and Frank, 2016b) (Collins and Frank, 2013;Whittington et al, 2020;Baram et al, 2021). Indeed, abstracting the action sequence structure of the Thunderstruck song may be useful for future learning.…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits

Calderon

Verguts

Frank

2021

Preprint

View full text Add to dashboard Cite

Adaptive sequential behavior is a hallmark of human cognition. In particular, humans can learn to produce precise spatiotemporal sequences given a certain context. For instance, musicians can not only reproduce learned action sequences in a context-dependent manner, they can also quickly and flexibly reapply them in any desired tempo or rhythm without overwriting previous learning. Existing neural network models fail to account for these properties. We argue that this limitation emerges from the fact that order information (i.e., the position of the action) and timing (i.e., the moment of response execution) are typically stored in the same neural network weights. Here, we augment a biologically plausible recurrent neural network of cortical dynamics to include a basal ganglia-thalamic module which uses reinforcement learning to dynamically modulate action. This associative cluster-dependent chain (ACDC) model modularly stores order and timing information in distinct loci of the network. This feature increases computational power and allows ACDC to display a wide range of temporal properties (e.g., multiple sequences, temporal shifting, rescaling, and compositionality), while still accounting for several behavioral and neurophysiological empirical observations. Finally, we apply this ACDC network to show how it can learn the famous Thunderstruck song and then flexibly play it in a bossa nova rhythm without further training.

show abstract

Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems

Cited by 89 publications

References 85 publications

Replay in minds and machines

Replay in minds and machines

Complementary task representations in hippocampus and prefrontal cortex for generalising the structure of problems

Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits

Contact Info

Product

Resources

About