Learning optimal decisions with confidence

Abstract

Diffusion decision models (DDMs) are immensely successful models for decision making under uncertainty and time pressure. In the context of perceptual decision making, these models typically start with two input units, organized in a neuron–antineuron pair. In contrast, in the brain, sensory inputs are encoded through the activity of large neuronal populations. Moreover, while DDMs are wired by hand, the nervous system must learn the weights of the network through trial and error. There is currently no normative theory of learning in DDMs and therefore no theory of how decision makers could learn to make optimal decisions in this context. Here, we derive such a rule for learning a near-optimal linear combination of DDM inputs based on trial-by-trial feedback. The rule is Bayesian in the sense that it learns not only the mean of the weights but also the uncertainty around this mean in the form of a covariance matrix. In this rule, the rate of learning is proportional (respectively, inversely proportional) to confidence for incorrect (respectively, correct) decisions. Furthermore, we show that, in volatile environments, the rule predicts a bias toward repeating the same choice after correct decisions, with a bias strength that is modulated by the previous choice’s difficulty. Finally, we extend our learning rule to cases for which one of the choices is more likely a priori, which provides insights into how such biases modulate the mechanisms leading to optimal decisions in diffusion models.

Reinforcement biases subsequent perceptual decisions when confidence is low, a widespread behavioral phenomenon

“…We show that this Bayesian on-line support vector machine also accounts for the observed choice bias strategy. Similarly, Bayesian learning in drift-diffusion models of decision making also makes similar predictions about confidence-dependent choice biases (Drugowitsch et al, 2019). Thus, either a sensory-based classification model modified to produce statistically optimal adjustments based on on-line feedback, or a reward-based model modified to account for the ambiguity in stimulus states produce broadly similar confidence-dependent trial-to-trial choice biases.…”

Confidence-controlled Hebbian learning efficiently extracts category membership from stimuli encoded in view of a categorization task

“…The adaptation of the coding layer has also been studied computationally [Bonnasse-Gahot and Nadal, 2008, Engel et al, 2015, Tajima et al, 2016, Min et al, 2020. Whereas most models of decision-making consider an uniform coding of the stimulus before the decision part [Beck et al, 2008, Drugowitsch et al, 2019, few models analyse the nature of a stimulus coding layer optimized in view of a categorization task [Bonnasse-Gahot and Nadal, 2008]. In agreement with the experimental findings, theoretical predictions with a two layers feedforward architecture give that tuning curves in the coding layer are sharper at the vicinity of a decision boundary.…”

“…The idea that confidence plays a role in learning has been proposed [Summerfield and De Lange, 2014, Meyniel and Dehaene, 2017, Meyniel, 2019. Within the framework of drift-diffusion models and taking a Bayesian viewpoint, [Drugowitsch et al, 2019] have shown that the optimal learning rate for categorization tasks should depend on the confidence in one's decision, where confidence is defined as the probability of having answered correctly. Within the attractor neural network framework, confidence in one's decision is well accounted for by the difference, at the time of decision, between the neural activities of the decision-specific neural pools [Wei andWang, 2015, Berlemont et al, 2020].…”

Rats strategically manage learning during perceptual decision making

“…Our estimation of SNR improvements during learning relies on the drift-diffusion model. Importantly, while this approach has been widely used in prior work [18,35,49,58], our conclusions are predicated on this model's approximate validity for our task. Future work could address this issue by using a paradigm in which learners with different response deadlines are tested at the same fixed response deadline, equalizing the impact of stimulus exposure at test.…”

“…However, it remains approximate and limited in several ways. The LDDM builds off the simplest form of a drift-diffusion model, and various extensions and related models have been proposed to better fit behavioral data, including urgency signals [47,[59][60][61][62], history-dependent effects [63][64][65][66][67][68][69], imperfect sensory integration [35], confidence [58,70,71], and multi-alternative choices [72,73]. More broadly, it remains unclear whether the drift-diffusion framework in fact underlies perceptual decision making, with a variety of other proposals providing differing accounts [38,74,75].…”