Causal evidence supporting the proposal that dopamine transients function as temporal difference prediction errors

Abstract

Reward-evoked dopamine transients are well-established as prediction errors. However the central tenet of temporal difference accounts-that similar transients evoked by reward-predictive cues also function as errors-remains untested. Here we addressed this by showing that optogenetically-shunting dopamine activity at the start of a reward-predicting cue prevents secondorder conditioning without affecting blocking. These results indicate that cue-evoked transients function as temporal-difference prediction errors rather than reward predictions.

Reinstatement of cortical outcome representations during higher-order learning

“…Using a second-order conditioning paradigm, we show associative transfer effects during higher-order conditioning, even when participants are unaware of the underlying associative structure of the experiment. Participants were more likely to select directly and indirectly appetitively paired stimuli over aversively paired stimuli, closely resembling rodent studies describing choice biases consistent with second-order conditioning (Maes et al, 2020; Sharpe et al, 2017). This suggests that humans, similar to rodents, implicitly acquire preferences through higher-order transfer learning mechanisms – a finding that goes beyond previous studies promoting acquisition of explicit associative relationships between stimuli (Jara et al, 2006; Pauli et al, 2019; Wang et al, 2020; Wimmer and Shohamy, 2012).…”

“…How the reinstatement of cortical US patterns found in our study relates to the observation in rats that midbrain dopamine neurons acquire temporal difference error signals in response to CS 2 (Maes et al, 2020) presents an important question for future studies. Furthermore, it would be of great interest to elucidate the directionality and exact content of information flow between lOFC and amygdala/anterior hippocampus, or between lOFC and medial OFC, and whether this transfer of information is supported by phase coherence in theta oscillations (Benchenane et al, 2010; Knudsen and Wallis, 2020; Young and Shapiro, 2011).…”

“…Since the “direct link” hypothesis of second-order conditioning is the only remaining explanatory attempt and has never been tested to date, the present study sought to provide evidence that CS 2 is directly paired with a neural representation of the US. Recent evidence suggests that acquisition of incentive properties by second-order stimuli (CS 2 ) is related to activity of mesencephalic dopamine neurons that encode a reward prediction error elicited by CS 1 (Maes et al, 2020). Presumably, SOC critically depends on convergence of neural patterns representing CS 2 and US in the amygdala (Gewirtz and Davis, 2000, 1997; Parkes and Westbrook, 2011; Setlow et al, 2002) and hippocampus (Gilboa et al, 2014).…”

Nigrostriatal Dopamine Signals Sequence-Specific Action-Outcome Prediction Errors

“…Many dopamine neurons signal errors in cued reward prediction (Houk et al, 1995;Montague et al, 1996;Schultz et al, 1997;Cohen et al, 2012;Eshel et al, 2015Eshel et al, , 2016Engelhard et al, 2019), i.e., any change in expectation of future reward or difference between actual versus expected reward predicted by the cues (Sutton and Barto, 2018). These dopaminergic prediction errors are thought to convey a teaching signal that is critical for multiple forms of associative learning across the corticostriatal topography (Yin et al, 2008;Balleine, 2019), spanning both classical Pavlovian stimulus-outcome conditioning (Flagel, et al, 2011;Steinberg et al, 2013;Chang et al, 2016;Saunders et al, 2018;Maes et al, 2020) and the formation of stimulus-response habits (Knowlton et al, 1996;Matsumoto et al, 1999;Faure et al, 2005;Belin and Everitt, 2008;Wang et al, 2011;Kim et al, 2015).…”

Tonic firing mode of midbrain dopamine neurons continuously tracks reward values changing moment-by-moment

“…An influential theory proposed that such phasic activity of dopamine neurons represents reward prediction errors, discrepancies between obtained and predicted reward values, in reinforcement learning (Doya, 2002;Montague et al, 1996;Schultz et al, 1997), and the reward-evoked phasic activity of dopamine neurons has been shown to regulate this type of learning (Chang et al, 2016;Stauffer et al, 2016;Steinberg et al, 2013;Tsai et al, 2009). In addition, the cue-evoked phasic activity of dopamine neurons has been reported to reflect the value of cued rewards (Matsumoto and Hikosaka, 2009;Roesch et al, 2007;Tobler et al, 2005), and has recently been demonstrated to influence behavior associated with cued rewards (Maes et al, 2020;Morrens et al, 2020).…”