Phasic Dopamine Release in the Rat Nucleus Accumbens Symmetrically Encodes a Reward Prediction Error Term

Abstract

Making predictions about the rewards associated with environmental stimuli and updating those predictions through feedback is an essential aspect of adaptive behavior. Theorists have argued that dopamine encodes a reward prediction error (RPE) signal that is used in such a reinforcement learning process. Recent work with fMRI has demonstrated that the BOLD signal in dopaminergic target areas meets both necessary and sufficient conditions of an axiomatic model of the RPE hypothesis. However, there has been no direct evidence that dopamine release itself also meets necessary and sufficient criteria for encoding an RPE signal. Further, the fact that dopamine neurons have low tonic firing rates that yield a limited dynamic range for encoding negative RPEs has led to significant debate about whether positive and negative prediction errors are encoded on a similar scale. To address both of these issues, we used fast-scan cyclic voltammetry to measure reward-evoked dopamine release at carbon fiber electrodes chronically implanted in the nucleus accumbens core of rats trained on a probabilistic decision-making task. We demonstrate that dopamine concentrations transmit a bidirectional RPE signal with symmetrical encoding of positive and negative RPEs. Our findings strengthen the case that changes in dopamine concentration alone are sufficient to encode the full range of RPEs necessary for reinforcement learning.

Adolescent Alcohol Exposure Amplifies the Incentive Value of Reward-Predictive Cues Through Potentiation of Phasic Dopamine Signaling

“…Consistent with previous reports (Hart et al, 2014;Tobler et al, 2005), phasic dopamine activity was modulated by reward size and expectation in both groups in a manner consistent with the reporting of a reward prediction error from formal models of reinforcement learning, with increased phasic dopamine signaling after better-than-expected outcomes and decreased phasic dopamine signaling after worse-thanexpected outcomes. Interestingly, our data showed that adolescent alcohol consumption promoted a greater sensitivity to the unexpected variation in reward sizes.…”

Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

“…This null result holds even at lower sample sizes (n ≅ 200 per category, randomly sampled). Prior work strongly supports the hypothesis that dopamine fluctuations in striatum should track RPEs (1,8,10,11,13,14). Our results contradict this expectation; however, the task we use was designed to also assess the impact of counterfactual feedback (e.g., difference between actual outcomes and what might have happened; Eq.…”

“…Experiments in animal models support the hypothesis that changes in dopamine release at target neural structures encode reward prediction errors (RPEs) (the difference between actual and expected outcomes) important for learning and value-based decision-making (1,(8)(9)(10)(11)(12). In support of this claim, direct recordings of spike activity in mesencephalic dopaminergic neurons in nonhuman primates demonstrate that these neurons encode prediction errors in future reward delivery (8-10, 13, 14) and they may also encode other computations relevant for reward-guided actions (1,(15)(16)(17).…”

Surprise! Dopamine signals mix action, value and error

“…Although the predominant evidence for RPEs comes from DA neuron electrophysiology, a few voltammetry studies, including that described above (albeit modulated by action), have largely provided converging support for these bidirectional signals 10 . However, voltammetry measurements exhibit slow drifts across time, and the documented phasic signals are hence always evaluated relative to the pre-cue baseline.…”

“…One possibility is that tonically active neurons, a cholinergic population that pause during windows of dopaminergic RPEs, may signal when to learn and can further enhance the contrast between tonic and phasic DA signals 13 , although this remains speculative. Finally, on a broader level, RPE theory has been so successful in part because it has generated clear and testable predictions, leading to experimental designs that parametrically manipulate factors that affect RPE with compelling results 14 , even satisfying an axiomatic description of RPEs independent of any specific implementation 10 . Moreover, RPE models can still show ramping under certain circumstances and depending on assumptions 15 , so some purists may prefer the parsimony of a single RPE mechanism until further evidence for value theory accrues.…”