Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

Abstract

The dopamine system is thought to be involved in making decisions about reward. Here we recorded from the ventral tegmental area in rats learning to choose between differently delayed and sized rewards. As expected, the activity of many putative dopamine neurons reflected reward prediction errors, changing when the value of the reward increased or decreased unexpectedly. During learning, neural responses to reward in these neurons waned and responses to cues that predicted reward emerged. Notably, this cue-evoked activity varied with size and delay. Moreover, when rats were given a choice between two differently valued outcomes, the activity of the neurons initially reflected the more valuable option, even when it was not subsequently selected.Dopamine is thought to be essential for reinforcement learning 1-4 . This hypothesis is rooted in single-unit studies, which show that the firing patterns of dopamine neurons accord well with a signal that reports errors in reward prediction 5-15 . This is true of firing for rewards as well as for cues that come to predict rewards. This signal is thought to regulate learning and to guide decision-making; however, few studies have investigated this issue in behavioral tasks that involve active decision-making 11,14 .To address this issue, we recorded single-unit activity in the ventral tegmental area (VTA) of rats performing a variant of a time-discounting task. It is known that animals prefer immediate over delayed reward (time discounting) 16-25 , but the role of dopamine in this impulsive behavior remains unclear 26-30 . We designed a task to isolate changes in neural activity related to reward size from changes in neural activity related to time to reward by independently varying when (after a short or long delay) or what (big or small) reward was to be delivered at the end of the trial. As the rats learned to respond to the more valuable odor cue, many dopamine neurons developed cue-selective activity during odor sampling, firing more strongly for cues that predicted either the more immediate reward or the larger one. Notably, these same neurons exhibited activity that was consistent with prediction errors during unexpected delivery or omission of reward. When the rats were given the opportunity to choose between two

The role of learning-related dopamine signals in addiction vulnerability

“…The phasic activation in response to CSs is independent of the sensory modality of the conditioned stimuli and increases with predicted reward magnitude (Bayer and Glimcher, 2005;Roesch et al, 2007;Tobler et al, 2005) and probability (Enomoto et al, 2011;Fiorillo et al, 2003;Morris et al, 2006;Nakahara et al, 2004;Satoh et al, 2003). This is again in line with the theoretical formulation as the expected value increases with the size of the reward and its probability.…”

“…This is again in line with the theoretical formulation as the expected value increases with the size of the reward and its probability. Furthermore, the longer the delay between the CS and the reward, the weaker the response (Fiorillo et al, 2008;Kobayashi and Schultz, 2008;Roesch et al, 2007), reflecting temporal discounting of future rewards. Finally, if a reward-predicting stimulus is itself preceded by another, earlier, stimulus, then the phasic activation of dopamine neurons transfers back to this earlier stimulus (Schultz et al, 1993), which is again captured by the above theoretical account (Montague et al, 1996) of model-free learning.…”

“…These data demonstrate that dopamine neurons compute the prediction-error term with respect to a quantitative and time-resolved expected total future reward term E[V(s t+1 |s t )]. (Morris et al, 2006;Roesch et al, 2007), and thus appear to be able to emit model-free prediction errors both when learning about stimulus values V MF (s) as in Pavlovian settings, and when learning about stimulusaction values Q MF (s, a) as in instrumental settings.…”

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

“…We make the common assumption that critic values are represented in ventral striatum and that phasic signals of dopamine convey the critic prediction error (Dayan & Daw, 2008;Montague et al, 1996;Roesch, Calu, & Schoenbaum, 2007). Actor learning.…”

“…It depends on the combined actor weight Act a , that is the weighted difference 1 This could be a single state s, or a state-action pair (s, a), following indications that prediction error could correspond to state-action prediction errors (Morris, Nevet, Arkadir, Vaadia, & Bergman, 2006;Roesch, Calu, & Schoenbaum, 2007).…”