Dopamine transients do not act as model-free prediction errors during associative learning

Abstract

Dopamine neurons are proposed to signal the reward prediction error in model-free reinforcement learning algorithms. This term represents the unpredicted or 'excess' value of the rewarding event, value that is then added to the intrinsic value of any antecedent cues, contexts or events. To support this proposal, proponents cite evidence that artificiallyinduced dopamine transients cause lasting changes in behavior. Yet these studies do not generally assess learning under conditions where an endogenous prediction error would occur. Here, to address this, we conducted three experiments where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquire value and instead entered into associations with the later events, whether valueless cues or valued rewards. These results show that in learning situations appropriate for the appearance of a prediction error, dopamine transients support associative, rather than model-free, learning.

Memory guidance of value-based decision making at an abstract level of representation

“…Previous research suggests that transfer from S2 to S1 in sensory preconditioning depends on the amount of learning during the association phase, initially increasing with the number of pairings and declining thereafter 27 , 28 . We thus assessed whether the amount of training affected memory-guided decision making and generalization.…”

Dopamine: The Neuromodulator of Long-Term Synaptic Plasticity, Reward and Movement Control

“…Recently, Maes et al, using inhibitory optogenetics to prevent cue-evoked DA signals found that the DA neuron activity observed in response to a reward-predictive cue is a prediction error, not a signal about the value of the cue [ 93 ]. Experiments using excitatory optogenetics to stimulate DA neuron activity published [ 94 ] confirmed that DA stimulation supports associative learning on antecedent cues without evidence of a cached value being ascribed to the cue. Additionally, Morrens et al [ 95 ] showed that novel, but not familiar, cues evoke DA release and that if DA release is inhibited during a novel cue, learning about that cue is impaired.…”

The role of executive function in shaping reinforcement learning

“…Most RL research since has focused on simple forms of learning from outcomes that act as primary or secondary rewards, such as food, money, or numeric points in a game. However, the path to an RPE is not always so straightforward: For instance, recent work departs from the role of dopaminergic signaling in standard RPEs based on scalar rewards, extending the domain of RL to learning from indirect experiences (e.g., secondary conditioning) and more abstract learning of associations based on sensory features [57,58]. These findings suggest that RL value computations integrate information beyond primary and secondary rewards.…”