Model-based hierarchical reinforcement learning and human action control

Abstract

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.

The algorithmic anatomy of model-based evaluation

“…Nevertheless, heuristic decision tree pruning has recently arisen as an important feature of human behaviour also in modern RL tasks [129], and other insights from this earlier work are starting to permeate modern notions of hierarchical control [56]. It thus seems a ripe time to revisit these models and results, and attempt to understand how they can be made to relate to the theoretical and experimental phenomena reviewed here.…”

“…Similarly, rather than entire transition matrices, individual sets of state transitions can also be aggregated. This arises mainly in the control case, where the set of one-step actions can be augmented by aggregate actions, known as options, which constitute an extended policy [56,57]. Again, an option includes both transition and reward models that aggregate the consequences of the extended action sequence.…”

Strategy selection as rational metareasoning.

“…This extension would correspond to learning option models (Sutton et al, 1999)-a form of model-based hierarchical reinforcement learning (Barto & Mahadevan, 2003;Sutton et al, 1999) that holds promise for explaining the complex hierarchical structure of human behavior (Botvinick & Weinstein, 2014). Both extensions could be combined with ideas from hierarchical reinforcement learning to capture how people discover novel, more effective strategies by flexibly combining elementary operations with partial.…”

The principles of goal-directed decision-making: from neural mechanisms to computation and robotics

“…habitual) systems?' The Special Issue includes three contributions that provide accessible introductions to the key neural and behavioural [32] and computational [26,28] signatures of goal-directed systems and compare them with alternatives (e.g. habitual) systems.…”

“…habitual) systems. The Special Issue also revolves around distinct computational schemes that are proposed to define goal-directed choice in a normative framework, which include model-based reinforcement learning [26], hierarchical reinforcement learning [28], ideomotor action [29], active inference [27] and game-theoretic approaches in the social domain [33]. Clearly, these and other proposals remain to be assessed empirically, and an important open question is whether they apply to the wide research field in which goal-directed choice is at play: from simple laboratory studies to complex ecological scenarios [39] and malfunctioning [38].…”