Learning optimal decisions with confidence

Abstract

Diffusion decision models are immensely successful models for human and animal decisions under uncertainty. The decisions they model require the temporal accumulation of evidence to improve choice accuracy, and thus balance the trade-off between the decisions' speed and their accuracy. Commonly, diffusion models have a one-dimensional abstract input that represents noisy momentary decision-related evidence. However, the nervous system typically uses population codes to represent sensory variables, which implies that the momentary evidence is distributed across multiple inputs. It is currently unknown how decision makers could learn how to combine these multiple inputs to obtain a one-dimensional abstract decision variable. We present here a Bayesian learning rule for learning a near-optimal linear combination of these inputs based on trial-by-trial feedback. The rule is Bayesian in the sense that it learns not only the mean of the weights but also the uncertainty around this mean in the form of a covariance matrix. This yields a rule whose learning rate is strongly modulated by decision confidence, providing a computational role of the latter for every-day decisions. Furthermore, we show that, in volatile environments, the rule predicts a bias towards repeating the same choice after correct decisions, with a bias strength that is modulated by the previous choice' difficulty. Last, we extend our learning rule to cases for which one of the choices is more likely a-priori, which provides new insights into how such biases modulate the mechanisms leading to optimal decisions in diffusion models.

Choice History Biases Subsequent Evidence Accumulation

“…This prediction was confirmed experimentally for explicit manipulations of prior probability of the choice options (Hanks et al, 2011). Indeed, within the framework of the DDM, this way of combining prior information with current evidence maximizes reward rate (Moran, 2015; see also Drugowitsch and Pouget, 2018). Only when evidence reliability is constant across trials should prior information be incorporated as a static bias (i.e., starting point).…”

The impact of learning on perceptual decisions and its implication for speed-accuracy tradeoffs

“…This strategy is close-to-optimal in the sense that it well-approximates the best possible category boundary estimate given all available information, and under the assumption that the "true" category boundary drifts stochastically across consecutive trials (see Experimental Procedures for details). Even though it approximates the optimal strategy, which is intractable, it yields behavioral performance indistinguishable from optimal (Drugowitsch and Pouget, 2018). The use of this strategy resulted in a diffusion-to-bound model with stimulus-dependent Bayesian learning (which we refer to as "Bayes-DDM") ( Fig.…”

“…Importantly, these functions were predicted rather than directly fit, since the only data used for the fits were the trialaveraged accuracy and RT curves. An important feature of the Bayes-DDM is that it depends on both the accumulated inputs and decision time , reflecting a form of decision confidence (Drugowitsch and Pouget, 2018;see Experimental Procedures). In tasks like ours, with a varying difficulty, harder trials are associated with later choices and come with a lower decision confidence (Kiani and Shadlen, 2009).…”

“…Here, we hypothesized that biases are in fact signatures of an optimal learning strategy that is adapted to natural environments, implying that they only appear "unwanted" (or suboptimal) in situations which those conditions do not hold (Summerfield, Behrens and Koechlin, 2011). Intuitively, an optimal learning agent must always use both priors (history of stimuli, choices and rewards) and current sensory information in proportion to their confidence (Pouget, Drugowitsch and Kepecs, 2016;Drugowitsch and Pouget, 2018). To test this idea formally, we derived the optimal choice policy and learning algorithm for an ideal observer that uses accumulation of evidence and reward statistics to infer choices and updates its stimuluschoice mapping-a Bayesian drift-diffusion model (Drugowitsch and Pouget, 2018).…”

“…Intuitively, an optimal learning agent must always use both priors (history of stimuli, choices and rewards) and current sensory information in proportion to their confidence (Pouget, Drugowitsch and Kepecs, 2016;Drugowitsch and Pouget, 2018). To test this idea formally, we derived the optimal choice policy and learning algorithm for an ideal observer that uses accumulation of evidence and reward statistics to infer choices and updates its stimuluschoice mapping-a Bayesian drift-diffusion model (Drugowitsch and Pouget, 2018).…”