Computational Evidence for Hierarchically-Structured Reinforcement Learning in Humans

Abstract

Humans have the fascinating ability to achieve goals in a complex and constantly changing world, still surpassing modern machine learning algorithms in terms of flexibility and learning speed. It is generally accepted that a crucial factor for this ability is the use of abstract, hierarchical representations, which employ structure in the environment to guide learning and decision making. Nevertheless, how we create and use these hierarchical representations is poorly understood. This study presents evidence that human behavior can be characterized as hierarchical reinforcement learning (RL). We designed an experiment to test specific predictions of hierarchical RL using a series of subtasks in the realm of context-based learning, and observed several behavioral markers of hierarchical RL, such as asymmetric switch costs between changes in higher-level versus lower-level features, faster learning in higher-valued compared to lower-valued contexts, and preference for higher-valued compared to lower-valued contexts. We replicated these results across three independent samples. We simulated three models: a classic RL, a hierarchical RL, and a hierar-chical Bayesian model, and compared their behavior to human results. While the flat RL model captured some aspects of participants’ sensitivity to outcome values, and the hierarchical Bayesian model some markers of transfer, only hierarchical RL accounted for all patterns observed in human behavior. This work shows that hierarchical RL, a biologically-inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments, and opens the avenue for future research in this field.

Computational evidence for hierarchically structured reinforcement learning in humans

“…Data Availability. All data for this study have been made available for only researchers through the National Institute of Mental Health Data Archive (60). Analysis and modeling code is available on GitHub (https://github.com/ MariaEckstein/TaskSets).…”

The role of executive function in shaping reinforcement learning

“…Learning to make this abstract choice may also involve RL, such that RL computations occur over two different state-action spaces in parallel -an abstract context and task-set space, and a more concrete stimulus-action space [51,52]. There is recent computational, behavioral, and neural evidence that stacked hierarchies of RL computations happen in parallel over more and more abstract types of states and choices, facilitating complex learning abilities [53,54,51]. Such learning may be supported by hierarchies of representations in prefrontal cortex [55,56].…”

Connecting Context-specific Adaptation in Humans to Meta-learning

“…Although it is well established that context-dependent adaptation is vital for flexible behavior, the computational mechanisms underlying how humans use contextual information to guide learning in a new situation are still poorly understood. While recent computational works have shed essential insights into understanding these mechanisms in simplified settings (Collins and Frank 2013;Eckstein and Collins 2020), we lack computational models that can be scaled up to more realistic tasks.…”