A systems-neuroscience model of phasic dopamine.

Abstract

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling.

The Role of the Lateral Habenula in Inhibitory Learning from Reward Omission

“…The omission of an expected reward seems to promote a similar backpropagation process to the one that takes place when a cue predicts reward. Interestingly, the LHb is likely to play a crucial role in this process (see also Mollick et al, 2020). As mentioned above, reward omission triggers a phasic depression in dopamine release.…”

“…Deceleration in the increase of associative strength with repeated experiences involving paired presentations of the stimulus and the reward would require an antagonistic mechanism. Inhibitory NAc-VTA projections have been hypothesized to play some role in said mechanism (see Mollick et al, 2020).…”

“…A clear advantage of these models is that they make physiological hypotheses more tenable (Frank, 2005). To our knowledge, only few recently proposed neural-network models (see Mollick et al, 2020;Vitay and Hamker, 2014) have incorporated nodes regarding the physiology of the LHb. Even so, those models do not account for the multifaceted circuitry (e.g., including the LHb-VTA-mPFC patway) of this structure which, we argue, is highly relevant for an utter understanding of its role in learning.…”

“…Omission of expected rewards at a precise time promotes dopamine dips in canonical midbrain neurons with a remarkable contribution of the LHb (see Mollick et al, 2020). These dopamine dips back-propagate to stimuli that consistently predict reward omission in a way that resembles plastic changes associated with dopamine release (Tobler et al, 2003).…”

Higher-Order Conditioning: What Is Learnt and How it Is Expressed

“…Higher-order conditioning procedures have become a popular means of examining the neurobiology of learning and memory (for a review, see Gewirtz and Davis, 2000; see also, e.g., Lin and Honey, 2011;Gilboa et al, 2014;Holland, 2016;Lay et al, 2018;Maes et al, 2020;Mollick et al, 2020). This popularity reflects the relevance of higher-order conditioning to clinical domains (e.g., Davey and Arulampalan, 1982;Davey and McKenna, 1983;Wessa and Flor, 2007; see also , Field, 2006;Haselgrove and Hogarth, 2011), but also the practical advantages of the procedures, and the potential insights that their use enables: The procedures allow the complex Note: A and X are conditioned stimuli and the US denotes an unconditioned stimulus.…”