A comparison of reinforcement learning models of human spatial navigation

Abstract

Reinforcement learning (RL) models have been influential in characterizing human learning and decision making, but few studies apply them to characterizing human spatial navigation and even fewer systematically compare RL models under different navigation requirements. Because RL can characterize one’s learning strategies quantitatively and in a continuous manner, and one’s consistency of using such strategies, it can provide a novel and important perspective for understanding the marked individual differences in human navigation and disentangle navigation strategies from navigation performance. One-hundred and fourteen participants completed wayfinding tasks in a virtual environment where different phases manipulated navigation requirements. We compared performance of five RL models (3 model-free, 1 model-based and 1 “hybrid”) at fitting navigation behaviors in different phases. Supporting implications from prior literature, the hybrid model provided the best fit regardless of navigation requirements, suggesting the majority of participants rely on a blend of model-free (route-following) and model-based (cognitive mapping) learning in such navigation scenarios. Furthermore, consistent with a key prediction, there was a correlation in the hybrid model between the weight on model-based learning (i.e., navigation strategy) and the navigator’s exploration vs. exploitation tendency (i.e., consistency of using such navigation strategy), which was modulated by navigation task requirements. Together, we not only show how computational findings from RL align with the spatial navigation literature, but also reveal how the relationship between navigation strategy and a person’s consistency using such strategies changes as navigation requirements change.

A Dynamic Bayesian Actor Model explains Endpoint Variability in Homing Tasks

“…A vast array of previous computational models of sequential goaldirected navigation behavior have been based on differing approaches, including artificial neural networks [68][69][70], reinforcement learning [65,[71][72][73], or a combination thereof [23]. Furthermore, individual modeling solutions can be distinguished by the respective navigation tasks considered, their complexity, the actions, and state information available to agents, which fundamentally influences the learning of internal representations, selflocalization, and planning.…”

“…Investigations involving model-free and model-based reinforcement learning in virtual mazes have yielded that human behavior is consistent with a model-based strategy [65], i.e., planning. More recent work explored the possibility that humans may use a hybrid strategy between model-free and model-based approaches [72][73][74]. However, in these maze environments, it is often assumed that there is no perceptual uncertainty about the state, i.e., the current state is fully observable, the actions available to the agent are a small set of discrete choices, and variability is associated only with learning across episodes or switching between model-based and model-free strategies.…”

g-distance: On the comparison of model and human heterogeneity

“…Although the existence of behavioral heterogeneity has long been appreciated (Ashby, Maddox, & Lee, 1994;Blyth, 1972;Estes, 1956;Merrell, 1931;Sidman, 1952;Siegler, 1987), it remains relevant today across numerous areas of inquiry. For example, heterogeneity of behavior has recently been reported in spatial navigation, where strategies vary substantially between individuals (He, Liu, Eschapasse, Beveridge, & Brown, 2022). It has also been reported in studies of stimulus generalization, where group-level generalization gradients appear to be an aggregate of at least two distinct generalization types -linear and peaked (Lee, Hayes, & Lovibond, 2018).…”

“…The presence of heterogeneity in human behavior, observed in our experiment, is not a one-off and might even be commonplace. For example, in just the last few years, heterogeneity has repeatedly been reported in human category and contingency learning experiments (He et al, 2022;Lee et al, 2018;Nosofsky & Hu, 2022). Where heterogeneity is present (or is suspected), there are some implications for empirical practice; the most obvious of which is the need for much larger samples than if behavior can be assumed to be homogenous.…”

Effects of Acute Stress on Rigid Learning, Flexible Learning, and Value-Based Decision-Making in Spatial Navigation

“…Most research investigating how acute stress affects learning focuses on how stress affects memory (Buchanan et al, 2006; Elzinga et al, 2005; Gagnon et al, 2019; Gagnon & Wagner, 2016; Guenzel et al, 2013; Kuhlmann et al, 2005; Schwabe et al, 2007, 2008; Smeets et al, 2007; Zoladz et al, 2011) and whether stress shifts the balance between habitual or rigid behaviors and cognitive map-like, flexible behavior (Brown et al, 2020; Brunyé et al, 2016; Park et al, 2017; Raio et al, 2020; Schwabe et al, 2007; Schwabe & Wolf, 2011; van Gerven et al, 2016). One implication from these literatures is that when we are faced with a decision between different strategies to meet a goal, to the extent that we derive values for the different strategies from memory (He, Liu, Beveridge, et al, 2022; He, Liu, Eschapasse, et al, 2022), stress may fundamentally change how our strategy decision reflects past experiences. Our primary goal in the present experiment was to test how stress affects different types of learning and how memory is used in subsequent decision-making.…”