Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal target

Parker, Nathan F.; Cameron, Courtney M.; Taliaferro, Joshua P; Lee, Junuk; Choi, Jung Yoon; Davidson, Thomas J.; Daw, Nathaniel D.; Witten, Ilana B.

doi:10.1038/nn.4287

Cited by 295 publications

(419 citation statements)

References 52 publications

(56 reference statements)

Supporting

Mentioning

356

Contrasting

Order By: Relevance

“…Considering the role of DA neuron burst firing in reward prediction error coding (Schultz et al, 1997; Glimcher, 2011), the observed effects of the mutation are to an extent unexpected, as no significant changes in learning rates were observed. Still, we note that the reduced win-stay probability is actually similar to the effect reported in the case of optogenetic studies, where the inhibition of DA neurons imitating negative reward prediction error reduced the likelihood of returning to the previously rewarded alternative (Hamid et al, 2016; Parker et al, 2016). Moreover, the study by Pessiglione et al (2006) offers a possible explanation for why the reduced bursting of DA neurons might have led to less deterministic behavior rather than a reduced learning rate.…”

Section: Discussionsupporting

confidence: 84%

Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making

Cieślak

Ahn

Bogacz

et al. 2018

eNeuro

View full text Add to dashboard Cite

show abstract

Section: Discussionsupporting

confidence: 84%

Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making

Cieślak

Ahn

Bogacz

et al. 2018

eNeuro

View full text Add to dashboard Cite

show abstract

“…Although dopaminergic activity cannot be assessed directly using fMRI, both effort and reward PEs were evident in segregated regions of dopaminergic-rich midbrain, and where an effective connectivity analysis indicated a directional influence from SN/VTA toward subcortical (reward PE) and cortical (effort PE) targets via ascending mesolimbic and mesocortical pathways, respectively. Dopaminergic midbrain is thought to comprise several distinct dopaminergic populations that have dissociable functions (54,56,67,68). Here we demonstrate a segregation between effort and reward learning within SN/VTA across the domains of task activation, functional connectivity, and gray matter density.…”

Section: Discussionmentioning

confidence: 99%

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Hauser

Eldar

Dolan

2017

Proc. Natl. Acad. Sci. U.S.A.

116

126

View full text Add to dashboard Cite

Optimal decision making mandates organisms learn the relevant features of choice options. Likewise, knowing how much effort we should expend can assume paramount importance. A mesolimbic network supports reward learning, but it is unclear whether other choice features, such as effort learning, rely on this same network. Using computational fMRI, we show parallel encoding of effort and reward prediction errors (PEs) within distinct brain regions, with effort PEs expressed in dorsomedial prefrontal cortex and reward PEs in ventral striatum. We show a common mesencephalic origin for these signals evident in overlapping, but spatially dissociable, dopaminergic midbrain regions expressing both types of PE. During action anticipation, reward and effort expectations were integrated in ventral striatum, consistent with a computation of an overall net benefit of a stimulus. Thus, we show that motivationally relevant stimulus features are learned in parallel dopaminergic pathways, with formation of an integrated utility signal at choice. effort prediction errors | reward prediction errors | apathy | substantia nigra/ventral tegmental area | dorsomedial prefrontal cortex O rganisms need to make energy-efficient decisions to maximize benefits and minimize costs, a tradeoff exemplified in effort expenditure (1-3). A key example occurs during foraging, where an overestimation of effort can lead to inaction and starvation (4), whereas underestimation of effort can result in persistent failure, as exemplified in the myth of Sisyphus (5).In a naturalistic environment, we often simultaneously learn about success in expending sufficient effort into an action as well as the reward we obtain from this same action. The reward outcomes that signal success and failure of an action are usually clear, although the effort necessary to attain success is often less transparent. Only by repeatedly experiencing success and failure is it possible to acquire an estimate of an optimal level of effort needed to succeed, without unnecessary waste of energy. This type of learning is important in contexts as diverse as foraging, hunting, and harvesting (6-8). Hull in his "law of less work" proposed that organisms "gradually learn" how to minimize effort expenditure (9). Surprisingly, we know little regarding the neurocognitive mechanisms that guide this form of simultaneous learning about reward and effort.A mesolimbic dopamine system encodes a teaching signal tethered to prediction of reward outcomes (10, 11). These reward prediction errors (PEs) arise from dopaminergic neurons in substantia nigra and ventral tegmental area (SN/VTA) and are broadcast to ventral striatum (VS) to mediate reward-related adaptation and learning (12, 13). Dopamine is also thought to provide a motivational signal (14-18), while dopaminergic deficits in rodents impair how effort and reward are arbitrated (1, 4, 19). The dorsomedial prefrontal cortex (dmPFC; spanning presupplementary motor area [pre-SMA] and dorsal anterior cingulate cortex [dACC]) is a candidate substrate...

show abstract

“…Dopamine modifies the efficacy of the corticostriatal synapses, as well as the striatal excitability directly, using signals that incorporate both expectation and external gains and losses (3)(4)(5). Decades of research have revealed the manner in which signals relaying information concerning expected and actual gains and costs are incorporated in the striatal dynamic system (6)(7)(8)(9).…”

mentioning

confidence: 99%

Pallidal spiking activity reflects learning dynamics and predicts performance

Schechtman

Noblejas

Mizrahi

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

The basal ganglia (BG) network has been divided into interacting actor and critic components, modulating the probabilities of different state-action combinations through learning. Most models of learning and decision making in the BG focus on the roles of the striatum and its dopaminergic inputs, commonly overlooking the complexities and interactions of BG downstream nuclei. In this study, we aimed to reveal the learning-related activity of the external segment of the globus pallidus (GPe), a downstream structure whose computational role has remained relatively unexplored. Recording from monkeys engaged in a deterministic three-choice reversal learning task, we found that changes in GPe discharge rates predicted subsequent behavioral shifts on a trial-by-trial basis. Furthermore, the activity following the shift encoded whether it resulted in reward or not. The frequent changes in stimulus-outcome contingencies (i.e., reversals) allowed us to examine the learning-related neural activity and show that GPe discharge rates closely matched across-trial learning dynamics. Additionally, firing rates exhibited a linear decrease in sequences of correct responses, possibly reflecting a gradual shift from goal-directed execution to automaticity. Thus, modulations in GPe spiking activity are highest for attentiondemanding aspects of behavior (i.e., switching choices) and decrease as attentional demands decline (i.e., as performance becomes automatic). These findings are contrasted with results from striatal tonically active neurons, which show none of these task-related modulations. Our results demonstrate that GPe, commonly studied in motor contexts, takes part in cognitive functions, in which movement plays a marginal role.basal ganglia | learning | attention | globus pallidus | actor-critic model T he basal ganglia have long been implicated in learning new skills and associations (1). One of the most influential models of these structures distinguishes between two domains: the main axis, whose function is to execute and choose between different actions based on their expected outcome, and the neuromodulators, which act to shape the connectivity within said axis by incorporating information regarding the results of actions on the internal and external state. These components are aptly named the actor and critic, respectively (2).The roles of the striatum, the largest structure in the main axis, and dopamine, the key neuromodulator serving as a critic in the basal ganglia framework, have been thoroughly studied. Dopamine modifies the efficacy of the corticostriatal synapses, as well as the striatal excitability directly, using signals that incorporate both expectation and external gains and losses (3-5). Decades of research have revealed the manner in which signals relaying information concerning expected and actual gains and costs are incorporated in the striatal dynamic system (6-9). However, these dopamine-and striato-centric views often fail to take into account our current understanding of the basal ganglia, which ac...

show abstract

Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal target

Cited by 295 publications

References 52 publications

Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making

Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Pallidal spiking activity reflects learning dynamics and predicts performance

Contact Info

Product

Resources

About