Neural basis of learning guided by sensory confidence and reward value

Abstract

Making efficient decisions requires combining present sensory evidence with previous reward values, and learning from the resulting outcome. To establish the underlying neural processes, we trained mice in a task that probed such decisions. Mouse choices conformed to a reinforcement learning model that estimates predicted value (reward value times sensory confidence) and prediction error (outcome minus predicted value). Predicted value was encoded in the pre-outcome activity of prelimbic frontal neurons and midbrain dopamine neurons. Prediction error was encoded in the post-outcome activity of dopamine neurons, which reflected not only reward value but also sensory confidence. Manipulations of these signals spared ongoing choices but profoundly affected subsequent learning. Learning depended on the pre-outcome activity of prelimbic neurons, but not dopamine neurons. Learning also depended on the post-outcome activity of dopamine neurons, but not prelimbic neurons. These results reveal the distinct roles of frontal and dopamine neurons in learning under uncertainty.

Lapses in perceptual decisions reflect exploration

“…This means that animals should continue to use the outcomes of previous trials to update the values of different actions as long as this uncertainty persists. Such persistent learning has been observed in a number of studies (Busse et al, 2011; Lak et al, 2018; Mendonca et al, 2018; Odoemene et al, 2018; Pinto et al, 2018; Scott et al, 2015). The uncertainty-dependent exploration model predicts that changes in action values should manifest as changes in lapse rates.…”

“…The subjects’ task is further complicated by perceptual uncertainty - on trials where the stimulus category is not fully known, credit cannot be unambiguously assigned to one stimulus-action pair when rewards are obtained (Lak et al, 2018). This predicts that conditions with higher perceptual uncertainty (e.g.…”

“…Similarly, outcomes on a given trial can influence decisions about stimuli on subsequent trials through reinforcement learning, giving rise to serial biases. These biases occur even though the ideal observer should treat the evidence on successive trials as independent (Busse et al, 2011; Lak et al, 2018; Mendonca et al, 2018). When subjects can control how long they sample the stimulus, subjects maximizing reward rate may choose to make premature decisions, sacrificing accuracy for speed (Bogacz et al, 2006; Drugowitsch, DeAngelis, et al, 2014).…”

“…An open question that remains is how the brain might tune the degree of exploration in proportion to uncertainty. An intriguing candidate for this is dopamine, whose phasic responses have been shown to reflect state uncertainty (Starkweather et al, 2017; Babayan, Uchida, and Gershman, 2018; Lak et al, 2018), and whose tonic levels have been shown to modulate exploration in mice on a lever-press task (Beeler et al, 2010), and context-dependent song variability in songbirds (Leblois, Wendel, and Perkel, 2010). Dopaminergic genes have been shown to predict individual differences in uncertainty-guided exploration in humans (Frank et al, 2009), and dopaminergic disorders such as Parkinson’s disease have been shown to disrupt the uncertainty-dependence of lapses across conditions on a multisensory task (Bertolini et al, 2015).…”

Control of adaptive action selection by secondary motor cortex during flexible visual categorization

“…Given the possibility that the mice could solve the task by updating separate value functions of different choices rather than comparing the stimulus with a decision criterion, we also designed a reinforcement learning (RL) model (Sutton and Barto, 1998) with sensory evidence (Lak et al, 2018). In this model, the expected values of left and right choices ( Ql and Qr ) are mapped into the mice’s choice through a softmax function: where Pr represents the probability of right choice.…”