The temporal precision of reward prediction in dopamine neurons

Abstract

Midbrain dopamine neurons are activated when reward is greater than predicted, and this error signal could teach target neurons both the value of reward and when it will occur. We used the dopamine error signal to measure how the expectation of reward was distributed over time. Animals were trained with fixed-duration intervals of 1-16 s between conditioned stimulus onset and reward. In contrast to the weak responses that have been observed after short intervals (1-2 s), activations to reward increased steeply and linearly with the logarithm of the interval. Results with varied stimulus-reward intervals suggest that the neural expectation was substantial after just half an interval had elapsed. Thus, the neural expectation of reward in these experiments was not highly precise and the precision declined sharply with interval duration. The neural precision of expectation appeared to be at least qualitatively similar to the precision of anticipatory licking behavior.

Absence of NMDA receptors in dopamine neurons attenuates dopamine release but not conditioned approach during Pavlovian conditioning

“…For example, unlike in our task in which the US delivery occurred during the last second of CS presentation, the CS in previous studies terminated shortly before reward delivery (4, 9); thus, CS termination just before reward delivery could provide a proximal cue that may have allowed the rats to predict the time of reward delivery more precisely, resulting in a smaller error in reward prediction. Likewise, in studies using monkeys in which the US occurred during the last second of CS presentation, US-elicited activation of dopamine neurons was maintained after training and remained larger than CS responses (35,36). This interpretation agrees with the observation that the mice in our paradigm were unable to time reward delivery as well as the same strain of mice trained in a paradigm with no overlap between the CS and US (26,27).…”

“…The persistence of the dopamine response to the US and the observation that it remains larger than the response to the CS is inconsistent with some previous reports (4,8,9) and correlates with the propensity of these mice to engage in goal-tracking as opposed to sign-tracking behavior (1,4,9). However, task parameters also can influence the pattern of CA behavior (34) and phasic dopamine transmission (35). For example, unlike in our task in which the US delivery occurred during the last second of CS presentation, the CS in previous studies terminated shortly before reward delivery (4, 9); thus, CS termination just before reward delivery could provide a proximal cue that may have allowed the rats to predict the time of reward delivery more precisely, resulting in a smaller error in reward prediction.…”

The role of learning-related dopamine signals in addiction vulnerability

“…This is again in line with the theoretical formulation as the expected value increases with the size of the reward and its probability. Furthermore, the longer the delay between the CS and the reward, the weaker the response (Fiorillo et al, 2008;Kobayashi and Schultz, 2008;Roesch et al, 2007), reflecting temporal discounting of future rewards. Finally, if a reward-predicting stimulus is itself preceded by another, earlier, stimulus, then the phasic activation of dopamine neurons transfers back to this earlier stimulus (Schultz et al, 1993), which is again captured by the above theoretical account (Montague et al, 1996) of model-free learning.…”

Timing in reward and decision processes

“…Second, the dopamine response reflects temporal discounting, both at the time of the conditioned, reward-predicting stimulus and at the time of the reward itself [23,24]. The dopamine activation to the CS predicting a reward with a longer delay is weaker than the response to another CS predicting the same physical amount of reward after a shorter delay.…”