Learning To Minimize Efforts versus Maximizing Rewards: Computational Principles and Neural Correlates

Abstract

The mechanisms of reward maximization have been extensively studied at both the computational and neural levels. By contrast, little is known about how the brain learns to choose the options that minimize action cost. In principle, the brain could have evolved a general mechanism that applies the same learning rule to the different dimensions of choice options. To test this hypothesis, we scanned healthy human volunteers while they performed a probabilistic instrumental learning task that varied in both the physical effort and the monetary outcome associated with choice options. Behavioral data showed that the same computational rule, using prediction errors to update expectations, could account for both reward maximization and effort minimization. However, these learning-related variables were encoded in partially dissociable brain areas. In line with previous findings, the ventromedial prefrontal cortex was found to positively represent expected and actual rewards, regardless of effort. A separate network, encompassing the anterior insula, the dorsal anterior cingulate, and the posterior parietal cortex, correlated positively with expected and actual efforts. These findings suggest that the same computational rule is applied by distinct brain systems, depending on the choice dimension-cost or benefit-that has to be learned.

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

“…In relation to dmPFC, activity increased if an effort threshold was higher than expected and was attenuated if it was lower than expected, suggestive of an invigorating function for future action. This finding is also in keeping with previous work on effort outcome (24,25), and a significant influence of dmPFC activity on subsequent change in effort execution [effect size: 0.04 ± 0.07; t(27) = 2.82, P = 0.009] supports its behavioral relevance in this task and is consistent with updating a subject's expectation about future effort requirements (33). Interestingly, dmPFC area processing effort PE peaks anterior to pre-SMA and lies anterior to where anticipatory effort signals are found in SMA (SI Appendix, Fig.…”

“…Critically, we demonstrate that both types of PE at outcome satisfy requirements for a full PE, representing both an effect of expectation and outcome (cf. 42), and thus extend previous single-attribute learning studies for reward PE (e.g., 12, 36, 42) and effort outcomes (24,25).…”

“…A first comparison revealed that the PE learning model outperformed nonlearning models where reward or effort was fixed rather than dynamically adapted, supporting the idea that subjects simultaneously learned about, and adjusted, their behavior to both features. We also found that the effort PE model outperformed a heuristic model that only adjusted its expectation based on success but did not scale the magnitude of adjustment using an effort PE, in line with a previous study showing a PE-like learning of effort (24). In addition, we compared the model with an optimal reward learning model, which tracks the previous reward magnitude and ignores the probabilistic null outcomes, revealing that PE-based reward learning outperformed this optimal model.…”

“…24,70) with effort discounting (39)(40)(41). The effort PE encoded in dmPFC can be seen as representing an adjustment in belief about the height of a required effort threshold.…”

“…The dorsomedial prefrontal cortex (dmPFC; spanning presupplementary motor area [pre-SMA] and dorsal anterior cingulate cortex [dACC]) is a candidate substrate for effort learning. For example, selective lesioning of this region engenders a preference for loweffort choices (15,(20)(21)(22)(23), while receiving effort feedback elicits responses in this same region (24,25). The importance of dopamine to effort learning is also hinted at in disorders with putative aberrant dopamine function, such as schizophrenia (26), where a symptom profile ("negative symptoms") often includes a lack of effort expenditure and apathy (27-30).…”