Decision-making invokes two fundamental axes of control: affect or valence, spanning reward and punishment, and effect or action, spanning invigoration and inhibition. We studied the acquisition of instrumental responding in healthy human volunteers in a task in which we orthogonalized action requirements and outcome valence. Subjects were much more successful in learning active choices in rewarded conditions, and passive choices in punished conditions. Using computational reinforcement-learning models, we teased apart contributions from putatively instrumental and Pavlovian components in the generation of the observed asymmetry during learning. Moreover, using model-based fMRI, we showed that BOLD signals in striatum and substantia nigra/ventral tegmental area (SN/VTA) correlated with instrumentally learnt action values, but with opposite signs for go and no-go choices. Finally, we showed that successful instrumental learning depends on engagement of bilateral inferior frontal gyrus. Our behavioral and computational data showed that instrumental learning is contingent on overcoming inherent and plastic Pavlovian biases, while our neuronal data showed this learning is linked to unique patterns of brain activity in regions implicated in action and inhibition respectively.

Pavlovian biases influence learning and decision making by intricately coupling reward seeking with action invigoration and punishment avoidance with action suppression. This bias is not always adaptive; it can oftentimes interfere with instrumental requirements. The prefrontal cortex is thought to help resolve such conflict between motivational systems, but the nature of this control process remains unknown. EEG recordings of mid-frontal theta band power are sensitive to conflict and predictive of adaptive control over behavior, but it is not clear whether this signal would reflect control over conflict between motivational systems. Here we utilized a task that orthogonalized action requirements and outcome valence while recording concurrent EEG in human participants. By applying a computational model of task performance, we derived parameters reflective of the latent influence of Pavlovian bias and how it was modulated by mid-frontal theta power during motivational conflict. Between subjects, individuals who performed better under Pavlovian conflict exhibited higher mid-frontal theta power. Within subjects, trial-to-trial variance in theta power was predictive of ability to overcome the influence of the Pavlovian bias, and this effect was most pronounced in individuals with higher mid-frontal theta to conflict. These findings demonstrate that mid-frontal theta is not only a sensitive index of prefrontal control, but it can also reflect the application of top-down control over instrumental processes.

The acquisition of reward and the avoidance of punishment could logically be contingent on either emitting or withholding particular actions. However, the separate pathways in the striatum for go and no-go appear to violate this independence, instead coupling affect and effect. Respect for this interdependence has biased many studies of reward and punishment, so potential action-outcome valence interactions during anticipatory phases remain unexplored. In a functional magnetic resonance imaging study with healthy human volunteers, we manipulated subjects' requirement to emit or withhold an action independent from subsequent receipt of reward or avoidance of punishment. During anticipation, in the striatum and a lateral region within the substantia nigra/ventral tegmental area (SN/VTA), action representations dominated over valence representations. Moreover, we did not observe any representation associated with different state values through accumulation of outcomes, challenging a conventional and dominant association between these areas and state value representations. In contrast, a more medial sector of the SN/VTA responded preferentially to valence, with opposite signs depending on whether action was anticipated to be emitted or withheld. This dominant influence of action requires an enriched notion of opponency between reward and punishment.

HighlightsPavlovian responses couple action and valence.This coupling interferes with instrumental learning and performance.Action dominates valence in the striatum and dopaminergic midbrain.Boosting dopamine enhances the dominance of action over valence in the striatum.Boosting dopamine decreases the extent of the behavioral coupling between action and valence.

Senescence affects the ability to utilize information about the likelihood of rewards for optimal decision-making. In a human functional magnetic resonance imaging (fMRI) study, we show that healthy older adults have an abnormal signature of expected value resulting in an incomplete reward prediction error signal in the nucleus accumbens, a brain region receiving rich input projections from substantia nigra/ventral tegmental area (SN/VTA) dopaminergic neurons. Structural connectivity between SN/VTA and striatum measured with diffusion tensor imaging (DTI) was tightly coupled to inter-individual differences in the expression of this expected reward value signal. The dopamine precursor levodopa (L-DOPA) increased the task-based learning rate and task performance in some older adults to a level shown by young adults. Critically this drug-effect was linked to restoration of a canonical neural reward prediction error. Thus we identify a neurochemical signature underlying abnormal reward processing in older adults and show this can be modulated by L-DOPA.

Subjects routinely control the vigor with which they emit motoric responses. However, the bulk of formal treatments of decision-making ignores this dimension of choice. A recent theoretical study suggested that action vigor should be influenced by experienced average reward rate and that this rate is encoded by tonic dopamine in the brain. We previously examined how average reward rate modulates vigor as exemplified by response times and found a measure of agreement with the first suggestion. In the current study, we examined the second suggestion, namely the potential influence of dopamine signaling on vigor. Ninety healthy subjects participated in a double-blind experiment in which they received one of the following: placebo, L-DOPA (which increases dopamine levels in the brain), or citalopram (which has a selective, if complex, effect on serotonin levels). Subjects performed multiple trials of a rewarded odd-ball discrimination task in which we varied the potential reward over time in order to exercise the putative link between vigor and average reward rate. Replicating our previous findings, we found that a significant fraction of the variance in subjects' responses could be explained by our experimentally manipulated changes in average reward rate. Crucially, this relationship was significantly stronger under L-Dopa than under Placebo, suggesting that the impact of average reward levels on action vigor is indeed subject to a dopaminergic influence.

No abstract

Invasive recording of dopamine neurons in the substantia nigra and ventral tegmental area (SN/VTA) of behaving animals suggests a role for these neurons in reward learning and novelty processing. In humans, functional magnetic resonance imaging (fMRI) is currently the only non-invasive event-related method to measure SN/VTA activity, but it is debated to what extent fMRI enables inference about dopaminergic responses within the SN/VTA. We consider the anatomical and functional parcellation of the primate SN/VTA and find that its homogeneity suggests little variation in the regional specificity of fMRI signals for reward-related dopaminergic responses. Hence, these responses seem to be well captured by the compound fMRI signal from the SN/VTA, which seems quantitatively related to dopamine release in positron emission tomography (PET). We outline how systematic investigation of the functional parcellation of the SN/VTA in animals, new developments in fMRI analysis and combined PET-fMRI studies can narrow the gap between fMRI and dopaminergic neurotransmission.