Behavioral and Neural Changes after Gains and Losses of Conditioned Reinforcers

Abstract

Human behaviors can be more powerfully influenced by conditioned reinforcers, such as money, than by primary reinforcers. Moreover, people often change their behaviors to avoid monetary losses. However, the effect of removing conditioned reinforcers on choices has not been explored in animals, and the neural mechanisms mediating the behavioral effects of gains and losses are not well understood. To investigate the behavioral and neural effects of gaining and losing a conditioned reinforcer, we trained rhesus monkeys for a matching pennies task in which the positive and negative values of its payoff matrix were realized by the delivery and removal of a conditioned reinforcer. Consistent with the findings previously obtained with non-negative payoffs and primary rewards, the animal's choice behavior during this task was nearly optimal. Nevertheless, the gain and loss of a conditioned reinforcer significantly increased and decreased, respectively, the tendency for the animal to choose the same target in subsequent trials. We also found that the neurons in the dorsomedial frontal cortex, dorsal anterior cingulate cortex, and dorsolateral prefrontal cortex often changed their activity according to whether the animal earned or lost a conditioned reinforcer in the current or previous trial. Moreover, many neurons in the dorsomedial frontal cortex also signaled the gain or loss occurring as a result of choosing a particular action as well as changes in the animal's behaviors resulting from such gains or losses. Thus, primate medial frontal cortex might mediate the behavioral effects of conditioned reinforcers and their losses.

Neuronal Reward and Decision Signals: From Theories to Data

“…Neurons in orbitofrontal and anterior cingulate cortex, and TANs in striatum, respond differentially to aversive airpuffs, reward liquids, and their predictive stimuli and follow reversal between punishment and reward (10,255,452,592). In dorsolateral prefrontal cortex, more neurons respond to reward than airpuff (283) and are differentially activated by gains or losses of reward tokens (532). In medial prefrontal cortex, dorsal neurons respond more frequently to air puffs, whereas ventral neurons are more sensitive to liquid reward (372).…”

“…Neuronal signals reflect the sum of positive and negative values from liquid or food rewards and aversive liquids or air puff punisher in monkey dopamine neurons (FIGURE 11D) (157,160) and anterior cingulate cortex (10), and from odor rewards in human orbitofrontal cortex (190). Compatible with coding negative value as component for Equation 25, some frontal cortical neurons are depressed by losses of gained rewards (532). Ideally, neuronal investigations would test EUsum with all constituent components, although this is impractical and experimenters usually investigate only a few of them at a time.…”

“…Neuronal activity combining expected reward value conjointly with object information is only reported from imperative trials in frontal and temporal lobe neurons, as mentioned above (231,319,371,394,411), but cannot be labeled as "chosen" value due to the nonchoice task nature. Neurons coding chosen value conjointly with action are found with free choices in dorsal and ventral striatum (275,306,418,479); globus pallidus (418); premotor, prefrontal, and anterior cingulate cortex (210,532,550,620); and parietal cortex (433). The actions are eye or hand movements towards different spatial targets in monkeys (210,306,418,433,494,550,571,620,643) and nose pokes and whole left versus right body movements in rats (275,479).…”

Temporally Dissociable Contributions of Human Medial Prefrontal Subregions to Reward-Guided Learning

“…dmPFC is likewise activated in fMRI studies of reinforcement learning and action selection (Behrens et al, 2007;Botvinick, 2007;Hare et al, 2011;Vickery et al, 2011;Kolling et al, 2012;Boorman et al, 2013;Economides et al, 2014). Combined with evidence from single-unit recording and lesion studies (Kennerley et al, 2006;Seo and Lee, 2009;Sheth et al, 2012), this has led to the idea that dmPFC subserves a role in learning from outcomes, as well as in deploying learnt values in action selection (Alexander and Brown, 2011;. Despite partially different functional connotations of vmPFC and dmPFC, both areas consistently tend to coactivate in fMRI studies of human decision making (FitzGerald et al, 2009;Hare et al, 2011;Nicolle et al, 2012;Boorman et al, 2013), which has rendered it difficult to understand the relation between these two regions within the decision making network.…”

Metacognitive monitoring during category learning: how success affects future behaviour

“…Previously, the influence of outcome reward had been shown to affect activity during a learning task (Seo, Barraclough, & Lee, 2007;Seo & Lee, 2009). Histed et al (2009) showed that a given trial's outcome had a direct effect on a trial-to-trial basis and that behaviour was altered solely on the basis of the previous trial's outcome (e.g., Seo, Barraclough, & Lee, 2007;Seo & Lee, 2009).…”