Model-based action planning involves cortico-cerebellar and basal ganglia networks

Abstract

Humans can select actions by learning, planning, or retrieving motor memories. Reinforcement Learning (RL) associates these processes with three major classes of strategies for action selection: exploratory RL learns state-action values by exploration, model-based RL uses internal models to simulate future states reached by hypothetical actions, and motor-memory RL selects past successful state-action mapping. In order to investigate the neural substrates that implement these strategies, we conducted a functional magnetic resonance imaging (fMRI) experiment while humans performed a sequential action selection task under conditions that promoted the use of a specific RL strategy. The ventromedial prefrontal cortex and ventral striatum increased activity in the exploratory condition; the dorsolateral prefrontal cortex, dorsomedial striatum, and lateral cerebellum in the model-based condition; and the supplementary motor area, putamen, and anterior cerebellum in the motor-memory condition. These findings suggest that a distinct prefrontal-basal ganglia and cerebellar network implements the model-based RL action selection strategy.

Compositional clustering in task structure learning

“…Note that the function mapping motor actions onto 99 cardinal movements can depend on environmental conditions, and thus, context (for example, wind 100 condition can change the relationship between primitive actions and movements in space for an 101 aerial drone). A similar mapping between arbitrary button presses and movements in the "finger 102 sailing" task has been used to provided evidence for model-based action planning in human 103 subjects [14,15]. Similarly, we can express the reward function in terms of cardinal movements 104 based on a location in space, R c (x, A) = Pr(r|x, A, c).…”

“…As a consequence, reusing policy components will 645 cause the agent to explore regions of the state space that have had high reward value in other 646 contexts, which as we have shown may or may not be an adaptive strategy. For example, successful 647 generalization in the "diabolical rooms" problem presented here, and the "finger sailing" task 648 presented by Fermin and colleagues [14,15], requires a separation of reward from movement 649 statistics. Indeed, the generalization of policy-dependent successor state representations works well 650 only under small deviations of the reward or transition function [10,38,39].…”

“…This can obviate planning, a challenge for any biological model of any model-based control. 688 Nonetheless, multiple lines of research suggest humans engage in model-based 689 control [14,15,39,45,46] and human subjects can re-use arbitrary action-movement mappings 690 (highly similar to the ones proposed here) for model-based control, suggesting a compositional 691 representation potentially mediated by the dorsolateral prefrontal cortex, dorsomedial striatum and 692 cerebellum [14,15]. 693 Finally, while we have presented independent clustering as motivated by human capabilities for 694 generalization, the question of whether human learning is better accounted for by independent or 695 joint clustering, or a mixture of the two, remains an open question.…”

“…13 For the purpose of clustering contexts, the mapping function was used as a component of the 14 likelihood function. Specifically, the likelihood function for the context-clustering assignments in 15 joint clustering is the product of the mapping and reward functions L(D|k) = k (a, A)R k (x, A). 16 Independent clustering assigns contexts into clusters separately for mappings and rewards, using 17 L(D|k ) = k (a, A) as the mapping cluster likelihood.…”

Predictive representations can link model-based reinforcement learning to model-free mechanisms

“…Some evidence suggests that distractor tasks at decision time have no effect on reward revaluation [27], consistent with SR-Dyna. Other recent work has demonstrated that humans benefit from additional pre-decision time in revaluation tasks that closely resemble "policy revaluation" [96] and that this benefit recruits a network including the prefrontal cortex and basal ganglia. Such work is consistent with the predictions of both Dyna-Q as well as SR-Dyna accounts of value updating presented here.…”