Participants preference for information as a function of time
Adapting to new environments is a hallmark of animal and human cognition, and Reinforcement Learning (RL) models provide a powerful and general framework for studying such adaptation. A fundamental learning component identified by RL models is that in the absence of direct supervision, when learning is driven by trial-and-error, exploration is essential. The necessary ingredients of effective exploration have been studied extensively in machine learning. However, the relevance of some of these principles to humans' exploration is still unknown. An important reason for this gap is the dominance of the Multi-Armed Bandit tasks in human exploration studies. In these tasks, the exploration component per se is simple, because local measures of uncertainty, most notably visit-counters, are sufficient to effectively direct exploration. By contrast, in more complex environments, actions have long-term exploratory consequences that should be accounted for when measuring their associated uncertainties. Here, we use a novel experimental task that goes beyond the bandit task to study human exploration. We show that when local measures of uncertainty are insufficient, humans use exploration strategies that propagate uncertainties over states and actions. Moreover, we show that the long-term exploration consequences are temporally-discounted, similar to the temporal discounting of rewards in standard RL tasks. Additionally, we show that human exploration is largely uncertainty-driven. Finally, we find that humans exhibit signatures of temporally-extended learning, rather than local, 1-step update rules which are commonly assumed in RL models. All these aspects of human exploration are well-captured by a computational model in which agents learn an exploration "value-function", analogous to the standard (reward-based) value-function in RL.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.