A causal role for right frontopolar cortex in directed, but not random, exploration

Zajkowski, Wojciech; Kossut, Małgorzata; Wilson, Robert

doi:10.1101/064741

Cited by 34 publications

(65 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As expected from the formal analysis, dopamine was found to specifically control the exploration parameter β (inverse temperature) which represents undirected exploration or random noise in the choice process converting values to actions [27], [31], [32], rather than directed exploration driven by uncertainty [5], [25], [26], [28].…”

Section: Discussionmentioning

confidence: 99%

“…Because dopamine is one of the key factors that may encode success or uncertainty, it might modulate decisions by biasing them toward options that present the largest uncertainty [25], [26]. This would correspond to a "directed" exploration strategy [5], [27], [28]. Alternatively, success and failure could affect tonic dopamine levels and control random exploration of all options, as recently proposed by Humphries et al (2012) [29].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dopamine regulates the exploration-exploitation trade-off in rats

Cinotti

Fresno

Aklil

et al. 2018

Preprint

View full text Add to dashboard Cite

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine controls this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple nonstationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision making when facing changing environmental contingencies.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Dopamine regulates the exploration-exploitation trade-off in rats

Cinotti

Fresno

Aklil

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Some of their applications involve modeling individual differences assuming participants are not completely independent (Pratte & Rouder, 2011), and predicting behavior of a new participant based on information of the population (Lee, 2018;Shiffrin et al, 2008). One simple way to express a hierarchy, is to set Gaussian distributions from which parameters are generated (Matzke, Dolan, Batchelder, & Wagenmakers, 2015;Zajkowski et al, 2017). Figure 3 is a graphical representation of this model.…”

Section: Hierarchical Delta-rule With Velocity Term (Hvd)mentioning

confidence: 99%

“…Some of their applications involve modeling individual differences assuming participants are not completely independent (Pratte & Rouder, 2011), and predicting behavior of a new subject based on the information of the population (Lee, 2018;Shiffrin et al, 2008). As it is frequently done given its simple interpretation for variability (Lee, 2018;Matzke, Dolan, Batchelder, & Wagenmakers, 2015;Zajkowski et al, 2017), we assumed that the higher-order distributions were Gaussian. Importantly, these distributions were set at the level of conditions implying that what is Standard delta-rule (SD).…”

Section: Learning Modelsmentioning

confidence: 99%

“…Kalman Filter. This model is a Bayesian form of SD which updates the learning rate on a trial-by-trial basis depending on the current level of uncertainty (Gershman, 2015(Gershman, , 2017Kakade & Dayan, 2002;Kalman, 1960;Navarro et al, 2018;Speekenbrink & Konstantinidis, 2015;Speekenbrink & Shanks, 2010;Zajkowski et al, 2017). The Kalman Filter estimates the mean of the Gaussian distribution generating the observations following:…”

Section: Learning Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Velocity estimation in reinforcement learning

Velázquez

Villarreal

Bouzas

2018

Preprint

View full text Add to dashboard Cite

AffiliationThe current work aims to study how people make predictions, under a reinforcement learning framework, in an environment that fluctuates from trial to trial and is corrupted with Gaussian noise. We developed a computer-based experiment where subjects were required to predict the future location of a spaceship that orbited around planet Earth. Its position was sampled from a Gaussian distribution with the mean changing at a variable velocity and four different values of variance that defined our signal-to-noise conditions. Three reinforcement learning algorithms using hierarchical Bayesian modeling were proposed as candidates to describe our data. The first and second models are the standard delta-rule and its Bayesian counterpart, the Kalman Filter. The third model is a deltarule incorporating a velocity component which is updated using prediction errors. The main advantage of the later over the first two is that it assumes participants estimate the trial-by-trial changes in the mean of the distribution generating the observations. We used leave-one-out cross-validation and the Widely Applicable Information Criterion to compare the predictive accuracy of the models. In general, our results provided evidence in favor of the model with the velocity term and showed that the learning rate of velocity and the decision noise change depending on the value of the signal-to-noise ratio. Finally, we modeled these changes using an extension of its hierarchical structure that allows us to make prior predictions for untested signal-to-noise conditions.

show abstract

Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges

Dennison

Sazhin

Smith

2022

WIRES Cognitive Science

View full text Add to dashboard Cite

In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field.

show abstract

A causal role for right frontopolar cortex in directed, but not random, exploration

Cited by 34 publications

References 29 publications

Dopamine regulates the exploration-exploitation trade-off in rats

Dopamine regulates the exploration-exploitation trade-off in rats

Velocity estimation in reinforcement learning

Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges

Contact Info

Product

Resources

About