Optimal policy for multi-alternative decisions

Tajima, Satohiro; Drugowitsch, Jan; Patel, Nisheet; Pouget, Alexandre

doi:10.1038/s41593-019-0453-9

Cited by 94 publications

(213 citation statements)

References 58 publications

Supporting

Mentioning

209

Contrasting

Order By: Relevance

“…First, they avoid a critical downside of binary decision tasks, which are biased by a confirmation/exclusion criterion, as the exclusion of one option inherently implies the only alternative available is the correct choice to make. Second, despite the significant increase in computational demands associated with 3-options tasks (Churchland et al, 2008; Churchland and Ditterich, 2012; Itthipuripat et al, 2018; Tajima et al, 2019), the tasks used for this study rely on a simple set of rules, easy-to-compute stochastic associations (80% for the most prevalent bead in a jar and for the most likely outcome value associated to a card) and only three types of discrete evidence (i.e. the three feedback values or the three bead colors), reducing the effect of second-order uncertainty (Fleming and Dolan, 2012; Fleming and Daw, 2017).…”

Section: Discussionmentioning

confidence: 99%

Similar network compositions, but distinct neural dynamics underlying belief updating in environments with and without explicit outcomes

Fiore¹,

Gu²

2019

Preprint

View full text Add to dashboard Cite

Beliefs about action-outcomes contingencies are often updated in opaque environments where feedbacks might be inaccessible and agents might need to rely on other information for evidence accumulation. It remains unclear, however, whether and how the neural dynamics subserving confidence and uncertainty during belief updating might be context-dependent. Here, we applied a Bayesian model to estimate uncertainty and confidence in healthy humans (n=28) using two multi-option fMRI tasks, one with and one without feedbacks. We found that across both tasks, uncertainty was computed in the anterior insular, anterior cingulate, and dorsolateral prefrontal cortices, whereas confidence was encoded in anterior hippocampus, amygdala and medial prefrontal cortex. However, dynamic causal modelling (DCM) revealed a critical divergence between how effective connectivity in these networks was modulated by the available information. Specifically, there was directional influence from the anterior insula to other regions during uncertainty encoding, independent of outcome availability. Conversely, the network computing confidence was driven either by the anterior hippocampus when outcomes were not available, or by the medial prefrontal cortex and amygdala when feedbacks were immediately accessible. These findings indicate that confidence encoding might largely rely on evidence accumulation and therefore dynamically changes as a function of the available sensory information (i.e. symbolic sequences monitored by the hippocampus, and monetary feedbacks computed by amygdala and medial prefrontal cortex). In contrast, uncertainty could be triggered by any information that disputes existing beliefs (i.e. processed in the insula), independent of its content.Significance StatementOur choices are guided by our beliefs about action-outcome contingencies. In environments where only one action leads to a desired outcome, high estimated action-outcome probabilities result in confidence, whereas low probabilities distributed across multiple choices result in uncertainty. These estimations are continuously updated, sometimes based on feedbacks provided by the environment, but sometimes this update takes place in opaque environments where feedbacks are not readily available. Here, we show that uncertainty computations are driven by the anterior insula, independent of feedback availability. Conversely, confidence encoding dynamically adapts to the information available, as we found it was driven either by the anterior hippocampus, when feedback was absent, or by the medial prefrontal cortex and amygdala, otherwise.

show abstract

Section: Discussionmentioning

confidence: 99%

Similar network compositions, but distinct neural dynamics underlying belief updating in environments with and without explicit outcomes

Fiore¹,

Gu²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The optimal stopping rule for animals facing certain kinds of perceptual and value-based decisions has been derived (Drugowitsch, Moreno-Bote, Churchland, Shadlen & Pouget, 2012; Tajima, Drugowitsch & Pouget, 2016; Tajima, Drugowitsch, Patel & Pouget, 2019). In the N -dimensional space defined by the N evidence accumulators, the optimal stopping policy is to terminate deliberation when the point in this space which represents the state of the N accumulators, reaches a time-dependent curved bound (Tajima et al, 2019; Fig. 1).…”

Section: Introductionmentioning

confidence: 99%

“…Each dimension represents the value of one accumulator x i . (B) The optimal stopping rule involves accumulating until a criterion on time, t , and the state of the N accumulators, x i , is met (Tajima, Drugowitsch, Patel & Pouget, 2019). Criterion shown only for purposes of illustration, and not derived.…”

Section: Introductionmentioning

confidence: 99%

“…In groundbreaking work, Tajima et al (2019) provided a potential solution to these issues by showing that approximately optimal stopping rules may be considerably simpler than optimal stopping rules. Specifically, they showed that if the state of the accumulators is mapped onto a curved manifold, approximately optimal stopping can be achieved by stopping when the value of the maximum mapped accumulator crosses a threshold.…”

Section: Introductionmentioning

confidence: 99%

“…Could the brain recycle representations of posterior probabilities (which are non-linear transformations of the accumulators), saving the need for additional neural hardware? How robust is the mapping used by Tajima et al (2019); must this transformation be adapted to task and environmental conditions?…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Rapid acquisition of near optimal stopping using Bayesian by-products

Calder-Travis

Slade

2019

Preprint

View full text Add to dashboard Cite

Prior to any decision, animals (including humans) must stop deliberating. Both accumulating evidence 8 for too little or too long can be costly. In contrast to accounts of decision making, accounts of stopping do not 9 typically claim that animals use Bayesian posteriors. Considering a generic perceptual decision making task 10 we show that, under approximation, only two variables are relevant to the question of when to stop evidence 11 accumulation; time and the maximum posterior probability. We explored the rate at which stopping rules are 12 learned using deep neural networks as model learners. A network which only used time and the maximum 13 posterior probability learned faster than any other network considered. Therefore, such an approach may be 14 highly adaptive, and animals may be able to reuse the same neural machinery they use for decisions for stopping. 15These results suggest that Bayesian inference may be even more important for animals than previously thought. 16

show abstract

Mixing memory and desire: How memory reactivation supports deliberative decision‐making

Wang

Feng

Bornstein

2021

WIRES Cognitive Science

View full text Add to dashboard Cite

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affect deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted on the basis of their uncertainty. We close by reviewing related findings and identifying areas for further research.

show abstract

Optimal policy for multi-alternative decisions

Cited by 94 publications

References 58 publications

Similar network compositions, but distinct neural dynamics underlying belief updating in environments with and without explicit outcomes

Similar network compositions, but distinct neural dynamics underlying belief updating in environments with and without explicit outcomes

Rapid acquisition of near optimal stopping using Bayesian by-products

Mixing memory and desire: How memory reactivation supports deliberative decision‐making

Contact Info

Product

Resources

About