Human Dorsal Striatal Activity during Choice Discriminates Reinforcement Learning Behavior from the Gambler's Fallacy

Abstract

Reinforcement learning theory has generated substantial interest in neurobiology, particularly because of the resemblance between phasic dopamine and reward prediction errors. Actor-critic theories have been adapted to account for the functions of the striatum, with parts of the dorsal striatum equated to the actor. Here, we specifically test whether the human dorsal striatum-as predicted by an actor-critic instantiation-is used on a trial-to-trial basis at the time of choice to choose in accordance with reinforcement learning theory, as opposed to a competing strategy: the gambler's fallacy. Using a partial-brain functional magnetic resonance imaging scanning protocol focused on the striatum and other ventral brain areas, we found that the dorsal striatum is more active when choosing consistent with reinforcement learning compared with the competing strategy. Moreover, an overlapping area of dorsal striatum along with the ventral striatum was found to be correlated with reward prediction errors at the time of outcome, as predicted by the actor-critic framework. These findings suggest that the same region of dorsal striatum involved in learning stimulus-response associations may contribute to the control of behavior during choice, thereby using those learned associations. Intriguingly, neither reinforcement learning nor the gambler's fallacy conformed to the optimal choice strategy on the specific decision-making task we used. Thus, the dorsal striatum may contribute to the control of behavior according to reinforcement learning even when the prescriptions of such an algorithm are suboptimal in terms of maximizing future rewards.

Hypoactivation of the Ventral and Dorsal Striatum During Reward and Loss Anticipation in Antipsychotic and Mood Stabilizer-Naive Bipolar Disorder

“…Coordinates for the VS were taken from meta-analytic data of previous functional magnetic resonance imaging (fMRI) studies of MID task performance (left: (Knutson and Greer, 2008), consistent with other recent studies using this version of the MID (eg, Balodis et al, 2012). The DS was defined using coordinates from published fMRI data indicating activation of the DS in response to a range of different rewarding stimuli (left: x ¼ À 9, y ¼ 3, z ¼ 15; right: x ¼ 9, y ¼ 3, z ¼ 15) (Valentin and O'Doherty, 2009), consistent with other fMRI studies focusing on the DS (eg, Jessup and O'Doherty, 2011).…”

Modulation of feedback related activity in the rostral anterior cingulate cortex during trial and error exploration

“…3; Supplemental material). These data confirm that the dorsal striatum participates in reward processing, and confirm its sensitivity to RPE signals (Amiez et al, 2012;Apicella et al, 2009Apicella et al, , 2011Brovelli et al, 2008;Haruno and Kawato, 2006b;Jessup and O'Doherty, 2011;Lau and Glimcher, 2007;Yamada et al, 2011;Zink et al, 2004). Interestingly, using a variant of the problem-solving task, Yamada et al reported that different populations of striatal cells code for positive feedback, for error feedback, for probability, and for search-repeat effect (Yamada et al, 2011).…”

Damage to insula abolishes cognitive distortions during simulated gambling

“…Anterior insula appears to represent risk predictions during choice, and risk prediction errors in response to decision outcomes (50). Such predictions about the uncertainty of the environment are relevant to the near-miss effect (51) and gambler's fallacy (35), and arguably less relevant to the illusion of control effect that did not vary across lesion groups here. Past work in cases with insula lesions has shown increases in risktaking and impaired discrimination between risky gains and risky losses (31,52).…”

“…This apparent inversion implies some systematic tendency in the insula group, but based on an alternative model of the task contingencies (and supported by different regions of the decision-making network). In a probabilistic environment, it is beneficial to select recently reinforced options, and a recent fMRI study indicated that dorsal striatal responses track reinforcement learning parameters in such a task (35). Other work highlights involvement of dorsolateral prefrontal cortex in detecting pattern violations (36) and switching responding after longer runs (37).…”