Influence of Reward Delays on Responses of Dopamine Neurons

Abstract

Psychological and microeconomic studies have shown that outcome values are discounted by imposed delays. The effect, called temporal discounting, is demonstrated typically by choice preferences for sooner smaller rewards over later larger rewards. However, it is unclear whether temporal discounting occurs during the decision process when differently delayed reward outcomes are compared or during predictions of reward delays by pavlovian conditioned stimuli without choice. To address this issue, we investigated the temporal discounting behavior in a choice situation and studied the effects of reward delay on the value signals of dopamine neurons. The choice behavior confirmed hyperbolic discounting of reward value by delays on the order of seconds. Reward delay reduced the responses of dopamine neurons to pavlovian conditioned stimuli according to a hyperbolic decay function similar to that observed in choice behavior. Moreover, the stimulus responses increased with larger reward magnitudes, suggesting that both delay and magnitude constituted viable components of dopamine value signals. In contrast, dopamine responses to the reward itself increased with longer delays, possibly reflecting temporal uncertainty and partial learning. These dopamine reward value signals might serve as useful inputs for brain mechanisms involved in economic choices between delayed rewards.

Chronic deep brain stimulation of the rat ventral medial prefrontal cortex disrupts hippocampal–prefrontal coherence

“…The most thorough examination of this frequency band in the behaving rodent may be that of Fujisawa and Buzsáki (2011), who found that coherence between 2 and 5 Hz increased between the ventral tegmental area (VTA) and prelimbic cortex during an odor-to-place matching task, and that this oscillation was phase-coupled with theta in the dorsal hippocampus. This report extended previous observations that VTA neurons often fire action potentials at regular intervals of 4 Hz (Hyland et al, 2002;Paladini and Tepper, 1999;Bayer et al, 2007;Dzirasa et al, 2010;Kobayashi and Schultz, 2008). Hippocampal-prefrontal coherence in the 4 Hz band can also increase when dopamine is injection directly into the prefrontal cortex (Benchenane et al, 2010).…”

Computational models of reinforcement learning: the role of dopamine as a reward signal

“…For instance, the magnitude of the change in DA neuron activity was recently shown to be modulated by the size of the reward prediction error and by the length of the stimulus-reward interval, such that the most valuable reward induced the greatest increase in neural activity (Kobayashi and Schultz 2008;Roesch et al 2007). This is in line with the TD model, which incorporates a discount factor to take these magnitude differences into account.…”

The role of learning-related dopamine signals in addiction vulnerability

“…This is again in line with the theoretical formulation as the expected value increases with the size of the reward and its probability. Furthermore, the longer the delay between the CS and the reward, the weaker the response (Fiorillo et al, 2008;Kobayashi and Schultz, 2008;Roesch et al, 2007), reflecting temporal discounting of future rewards. Finally, if a reward-predicting stimulus is itself preceded by another, earlier, stimulus, then the phasic activation of dopamine neurons transfers back to this earlier stimulus (Schultz et al, 1993), which is again captured by the above theoretical account (Montague et al, 1996) of model-free learning.…”