Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex

Abstract

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized—in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

Prefrontal Dynamics Associated with Efficient Detours and Shortcuts: A Combined Functional Magnetic Resonance Imaging and Magnetoencenphalography Study

“…Consistent with our caudate responses reflecting a model-based updating process, a previous fMRI study of navigation in a continually changing environment found that caudate activity correlated with parameters of a model-based representation of the environment (Simon & Daw, 2011;Dickinson & Balleine, 2002). More recently, caudate activity has been shown for events where pauses were implemented, as well as when there were unexpected changes in distance (Ribas-Fernandes, Shahnazian, Holroyd, & Botvinick, 2019).…”

The best laid plans: Computational principles of ACC

“…This model learned the values of different options (whether to navigate using an egocentric or allocentric frame of reference), where each option corresponded to sequences of low-level movements in the maze. This conceptualization aligns well with increasing evidence that ACC is sensitive to hierarchically-organized action representations [27] and reward signals [28,29]. It is also consistent with observations that ACC supports foraging behavior [30], which is naturally framed as a hierarchical problem.…”

Predictive Representations in Hippocampal and Prefrontal Hierarchies

“…These results support the hypothesis that prefrontal-hippocampal representations organize relational knowledge-in this case for spatial navigation-at different scales of generalization and abstraction (Behrens et al, 2018;Momennejad & Howard, 2018) . In the case of spatial navigation, this hierarchical representation in turn enables hierarchical planning and subgoal computation using graphs of the environment (Figure 1) abstracted at different scales (Ribas-Fernandes et al, 2018) . Our proposal is that planning at larger scales may be enabled by larger and more abstract scales of predictive representations in anterior PFC (Figure 1, large scale graph).…”