Generalization guides human exploration in vast decision spaces

Abstract

From foraging for food to learning complex games, many aspects of human behaviour can be framed as a search problem with a vast space of possible actions. Under finite search horizons, optimal solutions are generally unobtainable. Yet how do humans navigate vast problem spaces, which require intelligent exploration of unobserved actions? Using a variety of bandit tasks with up to 121 arms, we study how humans search for rewards under limited search horizons, where the spatial correlation of rewards (in both generated and natural environments) provides traction for generalization. Across a variety of different probabilistic and heuristic models, we find evidence that Gaussian Process function learning-combined with an optimistic Upper Confidence Bound sampling strategy-provides a robust account of how people use generalization to guide search. Our modelling results and parameter estimates are recoverable, and can be used to simulate human-like performance, providing insights about human behaviour in complex environments.All rights reserved. No reuse allowed without permission.was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint (which . http://dx.doi.org/10.1101/171371 doi: bioRxiv preprint first posted online Aug. 1, 2017; previous work exploring inductive biases in pure function learning contexts 21,22 and human behaviour in univariate function optimization 23 , we present a comprehensive approach using a robust computational modelling framework to understand how humans generalize in an active search task.Across three studies using uni-and bivariate multi-armed bandits with up to 121 arms, we compare a diverse set of computational models in their ability to predict individual human behaviour. In all experiments, the majority of subjects are best captured by a model combining function learning using Gaussian Process (GP) regression, with an optimistic Upper Confidence Bound (UCB) sampling strategy that directly balances expectations of reward with the reduction of uncertainty. Importantly, we recover meaningful and robust estimates about the nature of human generalization, showing the limits of traditional models of associative learning 24 in tasks where the environmental structure supports learning and inference.The main contributions of this paper are threefold:1. We introduce the spatially correlated multi-armed bandit as a paradigm for studying how people use generalization to guide search in larger problems space than traditionally used for studying human behaviour.2. We find that a Gaussian Process model of function learning robustly captures how humans generalize and learn about the structure of the environment, where an observed tendency towards undergeneralization is shown to sometimes be beneficial.3. We show that participants solve the exploration-exploitation dilemma by optimistically inflating expectations of reward by the underlying uncertainty, with recoverable evidence for the separate phenome...

Generalization and Search in Risky Environments

“…This model does not generalize over unseen arms at all, but rather only learns locally about the distribution of rewards for each option separately (Wu et al., in press). It can also be considered as a special case of the function learning model as the assumed correlation between points goes to zero.…”

Search as a simple take-the-best heuristic

“…Yet, research has mostly focused on search problems where exploration and exploitation happen simultaneously (e.g. MAB: [15]; abstract search: [16]; Lévy processes: [17]; comparison of different paradigms: [18]) and/or where jumps in the solution space are allowed (e.g., correlated MAB: [19]; sampling paradigm: [10]; secretary problem: [20]; random sampling: [21]). Nevertheless, many search problems are characterised by separated exploration and exploitation phases and gradual exploration; examples include animals deciding where to hunt prey [22,23], algorithms maximising their reward in a reinforcement learning settings [24], and humans visually searching for a lost item [25], or solving a complex problem [14,5].…”

Make‐or‐Break: Chasing Risky Goals or Settling for Safe Rewards?

“…An alternative to backward induction is to assume that the decision maker acts myopically and decides whether or not to pursue the make‐or‐break task further as if this decision were the last that they could make. Myopic strategies accurately describe human behavior in a wide array of settings, ranging from dynamic investment decisions and sequential hypothesis testing to sequential search and multi‐armed bandit tasks (see also Busemeyer & Rapoport, ; Gabaix, Laibson, Moloche, & Weinberg, ; Stojic, Analytis, & Speekenbrink, ; Thaler, Tversky, Kahneman, & Schwartz, ; Wu, Schulz, Speekenbrink, Nelson, & Meder, ; Zhang & Yu, ). In our case, as more time is allocated to the make‐or‐break task, some of the associated uncertainty becomes replaced by an actual outcome y = q m ( t ′) experienced up until time t ′ [0, T ].…”