A Bayesian analysis of human decision-making on bandit problems

Steyvers, Mark; Lee, Michael; Wagenmakers, Eric‐Jan

doi:10.1016/j.jmp.2008.11.002

Cited by 174 publications

(184 citation statements)

References 27 publications

(27 reference statements)

Supporting

Mentioning

181

Contrasting

Order By: Relevance

“…These are situations in which there are a fixed number of choice alternatives (e.g., two-arm bandits, like a slot machine in a casino) and each bandit has a fixed rate of reward, which is unknown to the decision maker. From trial to trial the decision-maker receives information (outcome-feedback/reward) from their choice between the alternatives, and their job is to reliably select sequentially from the alternatives so that they maximize their cumulative rewards [10]. The tasks not only involve discrete choices between two options [11,12], but can also include four [13] six [14], or even eight options [15].…”

Section: Bandit Tasksmentioning

confidence: 99%

Approaches to Learning to Control Dynamic Uncertainty

Osman

Glass

Hola

2015

Systems

View full text Add to dashboard Cite

Abstract:In dynamic environments, when faced with a choice of which learning strategy to adopt, do people choose to mostly explore (maximizing their long term gains) or exploit (maximizing their short term gains)? More to the point, how does this choice of learning strategy influence one's later ability to control the environment? In the present study, we explore whether people's self-reported learning strategies and levels of arousal (i.e., surprise, stress) correspond to performance measures of controlling a Highly Uncertain or Moderately Uncertain dynamic environment. Generally, self-reports suggest a preference for exploring the environment to begin with. After which, those in the Highly Uncertain environment generally indicated they exploited more than those in the Moderately Uncertain environment; this difference did not impact on performance on later tests of people's ability to control the dynamic environment. Levels of arousal were also differentially associated with the uncertainty of the environment. Going beyond behavioral data, our model of dynamic decision-making revealed that, in actual fact, there was no difference in exploitation levels between those in the highly uncertain or moderately uncertain environments, but there were differences based on sensitivity to negative reinforcement. We consider the implications of our findings with respect to learning and strategic approaches to controlling dynamic uncertainty.

show abstract

Section: Bandit Tasksmentioning

confidence: 99%

Approaches to Learning to Control Dynamic Uncertainty

Osman

Glass

Hola

2015

Systems

View full text Add to dashboard Cite

show abstract

“…Six of the participants reported that points earned was somehow a function of time or delay (the correct controlling variable), 2 reported that points were a function of the number of times chosen, 1 reported a complex geometrical relationship, and the remaining participants' reports were either vague or equivalent to reporting that they did not know. Sex, self-reported GPA, and self-reported strategy did not significantly predict the best-fitting value of θ, but our sample size was too small to identify all but the largest individual-difference effects (a prior study had found a weak negative correlation, r = −.09, between intelligence and exploratory behavior; Steyvers et al, 2009).…”

Section: Resultsmentioning

confidence: 64%

“…A player must explicitly explore an environment in order to learn the expected payoffs for these n options, and then can later exploit this knowledge. In a four-armed bandit task similar to the one used in the present study, Steyvers, Lee, and Wagenmakers (2009) employed a Bayesian optimal-decision model derived from the softmax equation (Luce, 1963) to explore how humans balance exploration with exploitation. In addition, eightstimulus arrays very similar to the one used in the present study have been used with nonhuman animals (Jensen, Miller, & Neuringer 2006) and humans (Rothstein, Jensen, & Neuringer 2008), and in both cases behavior came under the control of the prevailing contingencies.…”

Section: Task Descriptionmentioning

confidence: 99%

Pigeon and human performance in a multi-armed bandit task in response to changes in variable interval schedules

Racey¹,

Young²,

Garlick³

et al. 2011

Learn Behav

View full text Add to dashboard Cite

The tension between exploitation of the best options and exploration of alternatives is a ubiquitous problem that all organisms face. To examine this trade-off across species, pigeons and people were trained on an eight-armed bandit task in which the options were rewarded on a variable interval (VI) schedule. At regular intervals, each option's VI changed, thus encouraging dynamic increases in exploration in response to these anticipated changes. Both species showed sensitivity to the payoffs that was often well modeled by Luce's (1963) decision rule. For pigeons, exploration of alternative options was driven by experienced changes in the payoff schedules, not the beginning of a new session, even though each session signaled a new schedule. In contrast, people quickly learned to explore in response to signaled changes in the payoffs.

show abstract

“…Rather than the optimal fit, the marginal likelihood measures a model's average fit. Unfortunately, more than a decade after its introduction in psychology (but see Edwards, Lindman, & Savage, 1963, for an early example), there are only a handful of studies relying on the marginal likelihood for model evaluation (e.g., Lee, 2004;Massaro, Cohen, Campbell, & Rod riguez, 2001;Navarro & Lee, 2003;Steyvers, Lee, & Wagenmakers, 2009). Furthermore, we know of no study that has used the marginal likelihood to evaluate prototype and exemplar models against empirical data, let alone the intermediate models in the broader VAM family.…”

Section: Model Evaluation In Category Learningmentioning

confidence: 99%