Humans can select actions by learning, planning, or retrieving motor memories. Reinforcement Learning (RL) associates these processes with three major classes of strategies for action selection: exploratory RL learns state-action values by exploration, model-based RL uses internal models to simulate future states reached by hypothetical actions, and motor-memory RL selects past successful state-action mapping. In order to investigate the neural substrates that implement these strategies, we conducted a functional magnetic resonance imaging (fMRI) experiment while humans performed a sequential action selection task under conditions that promoted the use of a specific RL strategy. The ventromedial prefrontal cortex and ventral striatum increased activity in the exploratory condition; the dorsolateral prefrontal cortex, dorsomedial striatum, and lateral cerebellum in the model-based condition; and the supplementary motor area, putamen, and anterior cerebellum in the motor-memory condition. These findings suggest that a distinct prefrontal-basal ganglia and cerebellar network implements the model-based RL action selection strategy.

In this article, the authors examine whether and how humans use model-free, reflexive strategies and model-based, deliberative strategies in motor sequence learning. They asked subjects to perform the grid-sailing task, which required moving a cursor to different goal positions in a 5 × 5 grid using different key-mapping (KM) rules between 3 finger keys and 3 cursor movement directions. The task was performed under 3 conditions: Condition 1, new KM; Condition 2, new goal position with learned KM; and Condition 3, learned goal position with learned KM; with or without prestart delay time. The performance improvement with prestart delay was significantly larger under Condition 2. This result provides evidence that humans implement a model-based strategy for sequential action selection and learning by using previously learned internal model of state transition by actions.

Altruistic punishment following social norm violations promotes human cooperation. However, experimental evidence indicates that some forms of punishment are spiteful rather than altruistic. Using two types of punishment games and seven non-strategic games, we identified strong behavioural differences between altruistic and spiteful punishers. Altruistic punishers who rejected unfair offers in the ultimatum game and punished norm violators in the third-party punishment game behaved pro-socially in various non-strategic games. Spiteful punishers who rejected unfair offers in the ultimatum game but did not punish norm violators in the third-party punishment game behaved selfishly in non-strategic games. In addition, the left caudate nucleus was larger in spiteful punishers than in altruistic punishers. These findings are in contrast to the previous assumption that altruistic punishers derive pleasure from enforcement of fairness norms, and suggest that spiteful punishers derive pleasure from seeing the target experience negative consequences.

Social value orientations (SVOs) are economic preferences for the distribution of resources – prosocial individuals are more cooperative and egalitarian than are proselfs. Despite the social and economic implications of SVOs, no systematic studies have examined their neural correlates. We investigated the amygdala and dorsolateral prefrontal cortex (DLPFC) structures and functions in prosocials and proselfs by functional magnetic resonance imaging and evaluated cooperative behavior in the Prisoner’s Dilemma game. We found for the first time that amygdala volume was larger in prosocials and positively correlated with cooperation, while DLPFC volume was larger in proselfs and negatively correlated with cooperation. Proselfs’ decisions were marked by strong DLPFC and weak amygdala activity, and prosocials’ decisions were marked by strong amygdala activity, with the DLPFC signal increasing only in defection. Our findings suggest that proselfs’ decisions are controlled by DLPFC-mediated deliberative processes, while prosocials’ decisions are initially guided by automatic amygdala processes.