Learning offline: memory replay in biological and artificial reinforcement learning

Roscow, Emma L.; Chua, Raymond; Costa, Rui Ponte; Jones, Matt W.; Lepora, Nathan F.

doi:10.1016/j.tins.2021.07.007

Cited by 23 publications

(12 citation statements)

References 122 publications

(230 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, the opportunity cost hypothesis holds that cognitive effort occurs when the reward value of the current task under execution is less than the potential reward value of an alternative task, thereby promoting a shift to the other task (Agrawal et al, 2022;Boureau et al, 2015;Inzlicht et al, 2014;Kurzban et al, 2013). Cognitive effort could also induce pauses in cognitive activity that allow new memories to be consolidated in cortex (Agrawal et al, 2022;Holroyd & Verguts, 2021), consistent with computational arguments (Roscow et al, 2021) and biological (Holroyd & Verguts, 2021) and behavioral (Gershman et al, 2014;Wamsley, 2022) evidence of this.…”

Section: Summary and Discussionmentioning

confidence: 59%

The Controllosphere: The Neural Origin of Cognitive Effort

Holroyd¹

2023

Preprint

View full text Add to dashboard Cite

Why do some mental activities feel harder than others? The answer to this question is surprisingly controversial. Current theories propose that cognitive effort affords a computational benefit, such as instigating a switch from an activity with low reward value to a different activity with higher reward value. By contrast, in this article I relate cognitive effort to the fact that brain neuroanatomy and neurophysiology render some neural states more energy-efficient than others. I introduce the concept of the “controllosphere,” an energy-inefficient region of neural state space associated with high control, which surrounds the better-known “intrinsic manifold”, an energy-efficient subspace associated with low control. Integration of control-theoretic principles with classic neurocomputational models of cognitive control suggests that dorsolateral prefrontal cortex (DLPFC) implements a controller that can drive the system state into the controllosphere, anterior cingulate cortex (ACC) implements an observer that monitors changes of state of the controlled system, and cognitive effort reflects a mismatch between DLPFC and ACC energies for control and observation. On this account, cognitive effort scales with the energetic demands of the DLPFC control signal, especially when the consequences of the control are unobservable by ACC. Further, I propose that neural transitions through the controllosphere lead to a buildup of neural waste. Cognitive effort therefore prevents against neural damage by discouraging extended periods of high control.

show abstract

Section: Summary and Discussionmentioning

confidence: 59%

The Controllosphere: The Neural Origin of Cognitive Effort

Holroyd¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Several studies show that awake replay is required for the acquisition of a task rule that requires short-term memory (Jadhav et al 2012; Igata, Ikegaya, and Sasaki 2021; Fernández-Ruiz et al 2019). By extension, several computational models that have virtual agents learn a similar foraging task have significantly increased the learning performance by incorporating replay like activity (Mattar and Daw 2018; Hayes et al 2021; Roscow et al 2021; Johnson and Redish 2005). Learning a new task rule, however, is a complex iterative behavior that requires different brain functions, such as information registration, pattern inference, revision and goal-directed modulations, to coordinate together at different time scales.…”

Section: Discussionmentioning

confidence: 99%

Awake hippocampal replay is not required for short-term memory

Deceuninck

Kloosterman

2022

Preprint

View full text Add to dashboard Cite

Short-term memory (STM) on a time scale of seconds to minutes is required to successfully perform day-to-day tasks, for example when engaging in a meaningful conversation. Previous studies in both rodents and primates have correlated hippocampal cellular activity and behavioural expression of STM. This research has led to models describing the putative neural mechanism in the hippocampus that mediate STM. In these models, a key role has been given to hippocampal replay – reactivation of neurons representing a trajectory through space–but definitive causal evidence that can challenge or confirm the model is missing. In this study, we aimed to address the uncertainty around the role of awake replay in STM by collecting direct causal evidence from behaving rats. Signatures of replay events were detected in the hippocampus and disrupted using electrical stimulation of the ventral hippocampal commissure in rats that were trained on three different spatial memory tasks in a multi-arm radial maze. All tasks required memory of the recent past, but varied in the time scale over which information needed to be retained: (1) a multiple trial match-to sample task, (2) a single trial non-match to sample task and (3) a spatial sequence memory paradigm. Rats readily learned the task rules, but disruption of awake replay did not affect task performance or other behavioral measures in any of the task. Altogether, our results show for the first time with definitive causal evidence that awake replay is not required for STM of events or of their temporal order.

show abstract

“…The procedure was identical to Experiment 2a, except that participants were required to rest for at least 7 minutes (enforced as countdown on the screen) following the learning stage of the superordinate task phase. The rest period was included to promote memory replay and integration (Roscow et al, 2021).…”

Section: Methodsmentioning

confidence: 99%

Interference and integration in hierarchical task learning.

Lee¹,

Hazeltine²,

Jiang³

2022

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

A key feature of human task learning is shared task representation: Simple, subordinate tasks can be learned and then shared by multiple complex superordinate tasks as building blocks to facilitate task learning. An important yet unanswered question is how superordinate tasks sharing the same subordinate task affects the learning and memory of each other. Leveraging theories of associative memory, we hypothesize that shared subordinate tasks can cause both interference and facilitation between superordinate tasks. These hypotheses are tested using a novel experimental task which trains participants to perform superordinate tasks consisting of shared, trained subordinate tasks. Across 3 experiments, we demonstrate that sharing a subordinate task can (a) impair the memory of previously learned superordinate tasks and (b) integrate learned superordinate tasks to facilitate new superordinate task learning without direct experience. These findings shed light on the organizational principles of task knowledge and their consequences on task learning.

show abstract

Learning offline: memory replay in biological and artificial reinforcement learning

Cited by 23 publications

References 122 publications

The Controllosphere: The Neural Origin of Cognitive Effort

The Controllosphere: The Neural Origin of Cognitive Effort

Awake hippocampal replay is not required for short-term memory

Interference and integration in hierarchical task learning.

Contact Info

Product

Resources

About