The Good, the Bad, and the Irrelevant: Neural Mechanisms of Learning Real and Hypothetical Rewards and Effort

Scholl, Jacqueline; Kolling, Nils; Nelissen, Natalie; Wittmann, Marco K.; Harmer, Catherine J.; Rushworth, Matthew F. S.

doi:10.1523/jneurosci.0396-15.2015

Cited by 80 publications

(113 citation statements)

References 59 publications

Supporting

Mentioning

103

Contrasting

Order By: Relevance

“…In relation to dmPFC, activity increased if an effort threshold was higher than expected and was attenuated if it was lower than expected, suggestive of an invigorating function for future action. This finding is also in keeping with previous work on effort outcome (24,25), and a significant influence of dmPFC activity on subsequent change in effort execution [effect size: 0.04 ± 0.07; t(27) = 2.82, P = 0.009] supports its behavioral relevance in this task and is consistent with updating a subject's expectation about future effort requirements (33). Interestingly, dmPFC area processing effort PE peaks anterior to pre-SMA and lies anterior to where anticipatory effort signals are found in SMA (SI Appendix, Fig.…”

Section: Distinct Striatal and Cortical Representations Of Reward Andsupporting

confidence: 92%

“…Critically, we demonstrate that both types of PE at outcome satisfy requirements for a full PE, representing both an effect of expectation and outcome (cf. 42), and thus extend previous single-attribute learning studies for reward PE (e.g., 12, 36, 42) and effort outcomes (24,25).…”

Section: Discussionsupporting

confidence: 78%

“…Although this entails a difference in how feedback is presented, we consider it unlikely to influence neural processes, given that previous studies with balanced designs showed activations in remarkably similar regions to ours (e.g., 24,25) and because prediction error signals are generally insensitive to outcome modality (primary/secondary reinforcer, magnitude/probabilistic feedback, learning/no learning) (e.g., 12,36,70,77). Moreover, the specificity of the signals in VS and dmPFC for either reward or effort, including a modulation by expectation, favors a view that the pattern of responses observed in these regions reflects specific prediction error signals as opposed to simple feedback signals.…”

Section: Discussionmentioning

confidence: 53%

“…To examine the robustness of this double dissociation, we sampled activity from independently derived regions of interest (ROIs; VS derived from www.neurosynth.org; dmPFC derived from ref. 25), and again found a significant double dissociation in both VS (SI Appendix, Fig. S5A) and dmPFC (SI Appendix, Fig.…”

Section: Distinct Striatal and Cortical Representations Of Reward Andmentioning

confidence: 71%

“…The dorsomedial prefrontal cortex (dmPFC; spanning presupplementary motor area [pre-SMA] and dorsal anterior cingulate cortex [dACC]) is a candidate substrate for effort learning. For example, selective lesioning of this region engenders a preference for loweffort choices (15,(20)(21)(22)(23), while receiving effort feedback elicits responses in this same region (24,25). The importance of dopamine to effort learning is also hinted at in disorders with putative aberrant dopamine function, such as schizophrenia (26), where a symptom profile ("negative symptoms") often includes a lack of effort expenditure and apathy (27-30).…”

mentioning

confidence: 99%

See 4 more Smart Citations

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Hauser

Eldar

Dolan

2017

Proc. Natl. Acad. Sci. U.S.A.

120

127

View full text Add to dashboard Cite

Optimal decision making mandates organisms learn the relevant features of choice options. Likewise, knowing how much effort we should expend can assume paramount importance. A mesolimbic network supports reward learning, but it is unclear whether other choice features, such as effort learning, rely on this same network. Using computational fMRI, we show parallel encoding of effort and reward prediction errors (PEs) within distinct brain regions, with effort PEs expressed in dorsomedial prefrontal cortex and reward PEs in ventral striatum. We show a common mesencephalic origin for these signals evident in overlapping, but spatially dissociable, dopaminergic midbrain regions expressing both types of PE. During action anticipation, reward and effort expectations were integrated in ventral striatum, consistent with a computation of an overall net benefit of a stimulus. Thus, we show that motivationally relevant stimulus features are learned in parallel dopaminergic pathways, with formation of an integrated utility signal at choice. effort prediction errors | reward prediction errors | apathy | substantia nigra/ventral tegmental area | dorsomedial prefrontal cortex O rganisms need to make energy-efficient decisions to maximize benefits and minimize costs, a tradeoff exemplified in effort expenditure (1-3). A key example occurs during foraging, where an overestimation of effort can lead to inaction and starvation (4), whereas underestimation of effort can result in persistent failure, as exemplified in the myth of Sisyphus (5).In a naturalistic environment, we often simultaneously learn about success in expending sufficient effort into an action as well as the reward we obtain from this same action. The reward outcomes that signal success and failure of an action are usually clear, although the effort necessary to attain success is often less transparent. Only by repeatedly experiencing success and failure is it possible to acquire an estimate of an optimal level of effort needed to succeed, without unnecessary waste of energy. This type of learning is important in contexts as diverse as foraging, hunting, and harvesting (6-8). Hull in his "law of less work" proposed that organisms "gradually learn" how to minimize effort expenditure (9). Surprisingly, we know little regarding the neurocognitive mechanisms that guide this form of simultaneous learning about reward and effort.A mesolimbic dopamine system encodes a teaching signal tethered to prediction of reward outcomes (10, 11). These reward prediction errors (PEs) arise from dopaminergic neurons in substantia nigra and ventral tegmental area (SN/VTA) and are broadcast to ventral striatum (VS) to mediate reward-related adaptation and learning (12, 13). Dopamine is also thought to provide a motivational signal (14-18), while dopaminergic deficits in rodents impair how effort and reward are arbitrated (1, 4, 19). The dorsomedial prefrontal cortex (dmPFC; spanning presupplementary motor area [pre-SMA] and dorsal anterior cingulate cortex [dACC]) is a candidate substrate...

show abstract

Section: Distinct Striatal and Cortical Representations Of Reward Andsupporting

confidence: 92%

Section: Discussionsupporting

confidence: 78%

Section: Discussionmentioning

confidence: 53%

Section: Distinct Striatal and Cortical Representations Of Reward Andmentioning

confidence: 71%

mentioning

confidence: 99%

See 3 more Smart Citations

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Hauser

Eldar

Dolan

2017

Proc. Natl. Acad. Sci. U.S.A.

120

127

View full text Add to dashboard Cite

show abstract

Altered motivation of effortful decision‐making for self and others in subthreshold depression

Rong

Dong

Zheng

et al. 2022

Depression and Anxiety

View full text Add to dashboard Cite

Background: Amotivation is a typical feature in major depressive disorders and refers to individuals exhibiting reduced willingness to exert effort for rewards.However, the motivation pattern when deciding whether to exert effort for self versus others in people with depression remains unclear.Methods: We conducted a functional magnetic resonance imaging study and employed an adapted Effort-Expenditure for Rewards Task in subthreshold depressive (SD) participants (n = 33) and healthy controls (HC) (n = 32). This required participants to choose between a fixed low-effort/low-reward and a variable higheffort/high-reward option, and then immediately exert effort to obtain corresponding rewards for themselves or for unfamiliar people.Results: Compared with the HC group, the SD group showed blunted activity in the left dorsal anterior cingulate cortex/dorsomedial prefrontal cortex, bilateral anterior insula (AI), and right putamen-left dorsolateral prefrontal cortex functional connectivity when choosing to exert effort for themselves. Additionally, the SD group exhibited increased willingness and greater activation in the bilateral AI when choosing to exert effort for others. Furthermore, these brain activations and functional connectivity were positively related to self-reported motivation.Conclusions: These findings show altered motivation during effort-based decisionmaking in individuals with the mild depressive state, particularly with higher motivation for others. Thus, this suggests that motivational behaviors and prefrontal-striatal circuitry are altered in individuals with SD, which can be utilized to discover treatment targets and develop strategies to address mental illness caused by motivation disorders.

show abstract

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta‐analysis

Fouragnan

Retzler

Philiastides

2018

Human Brain Mapping

139

126

View full text Add to dashboard Cite

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

show abstract

The Good, the Bad, and the Irrelevant: Neural Mechanisms of Learning Real and Hypothetical Rewards and Effort

Cited by 80 publications

References 59 publications

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Altered motivation of effortful decision‐making for self and others in subthreshold depression

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta‐analysis

Contact Info

Product

Resources

About