Neuronal Encoding in Prefrontal Cortex during Hierarchical Reinforcement Learning

Chiang, Feng-Kuei; Wallis, Jonathan D.

doi:10.1162/jocn_a_01272

Cited by 9 publications

(4 citation statements)

References 34 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent research has started to support this possibility. In a temporal abstraction paradigm, reward prediction errors at two levels of abstraction were observed in human electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) (47; 46), as well as primate anterior cingulate firing patterns (13). Similarly, a state abstraction paradigm revealed neural value signals at two levels of abstraction in human BOLD (24).…”

Section: Introductionmentioning

confidence: 99%

Computational Evidence for Hierarchically-Structured Reinforcement Learning in Humans

Eckstein¹,

Collins²

2019

Preprint

View full text Add to dashboard Cite

Humans have the fascinating ability to achieve goals in a complex and constantly changing world, still surpassing modern machine learning algorithms in terms of flexibility and learning speed. It is generally accepted that a crucial factor for this ability is the use of abstract, hierarchical representations, which employ structure in the environment to guide learning and decision making. Nevertheless, how we create and use these hierarchical representations is poorly understood. This study presents evidence that human behavior can be characterized as hierarchical reinforcement learning (RL). We designed an experiment to test specific predictions of hierarchical RL using a series of subtasks in the realm of context-based learning, and observed several behavioral markers of hierarchical RL, such as asymmetric switch costs between changes in higher-level versus lower-level features, faster learning in higher-valued compared to lower-valued contexts, and preference for higher-valued compared to lower-valued contexts. We replicated these results across three independent samples. We simulated three models: a classic RL, a hierarchical RL, and a hierar-chical Bayesian model, and compared their behavior to human results. While the flat RL model captured some aspects of participants’ sensitivity to outcome values, and the hierarchical Bayesian model some markers of transfer, only hierarchical RL accounted for all patterns observed in human behavior. This work shows that hierarchical RL, a biologically-inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments, and opens the avenue for future research in this field.

show abstract

Section: Introductionmentioning

confidence: 99%

Computational Evidence for Hierarchically-Structured Reinforcement Learning in Humans

Eckstein¹,

Collins²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Further analyses confirmed that these effects did not result from perceptual-or motor-related task confounds. Since RF2011, the HRL framework has been adopted in numerous theoretical and experimental studies (Chiang & Wallis, 2018;Umemoto, HajiHosseini, Yates, & Holroyd, 2017;Balaguer, Spiers, Hassabis, & Summerfield, 2016;Zarr & Brown, 2016;Holroyd & McClure, 2015;Badre & Frank, 2012).…”

Section: Introductionmentioning

confidence: 99%

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex

Ribas-Fernandes

Shahnazian

Holroyd

et al. 2019

Journal of Cognitive Neuroscience

View full text Add to dashboard Cite

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized—in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

show abstract

“…where SSE reduces is the sum of square error due to predictor X , SSE full model . is the sum of square error of the full model with all predictors 3 .…”

Section: Methodsmentioning

confidence: 99%

Building compositional tasks with shared neural subspaces

Tafazoli,

Bouchacourt,

Ardalan

et al. 2024

Preprint

View full text Add to dashboard Cite

Cognition is remarkably flexible; we are able to rapidly learn and perform many different tasks. Theoretical modeling has shown artificial neural networks trained to perform multiple tasks will re-use representations and computational components across tasks. By composing tasks from these sub-components, an agent can flexibly switch between tasks and rapidly learn new tasks. Yet, whether such compositionality is found in the brain is unknown. Here, we show the same subspaces of neural activity represent task-relevant information across multiple tasks, with each task compositionally combining these subspaces in a task-specific manner. We trained monkeys to switch between three compositionally related tasks. Neural recordings found task-relevant information about stimulus features and motor actions were represented in subspaces of neural activity that were shared across tasks. When monkeys performed a task, neural representations in the relevant shared sensory subspace were transformed to the relevant shared motor subspace. Subspaces were flexibly engaged as monkeys discovered the task in effect; their internal belief about the current task predicted the strength of representations in task-relevant subspaces. In sum, our findings suggest that the brain can flexibly perform multiple tasks by compositionally combining task-relevant neural representations across tasks.

show abstract

Neuronal Encoding in Prefrontal Cortex during Hierarchical Reinforcement Learning

Cited by 9 publications

References 34 publications

Computational Evidence for Hierarchically-Structured Reinforcement Learning in Humans

Computational Evidence for Hierarchically-Structured Reinforcement Learning in Humans

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex

Building compositional tasks with shared neural subspaces

Contact Info

Product

Resources

About