Neural computations underlying action-based decision making in the human brain

Abstract

Action-based decision making involves choices between different physical actions to obtain rewards. To make such decisions the brain needs to assign a value to each action and then compare them to make a choice. Using fMRI in human subjects, we found evidence for action-value signals in supplementary motor cortex. Separate brain regions, most prominently ventromedial prefrontal cortex, were involved in encoding the expected value of the action that was ultimately taken. These findings differentiate two main forms of value signals in the human brain: those relating to the value of each available action, likely reflecting signals that are a precursor of choice, and those corresponding to the expected value of the action that is subsequently chosen, and therefore reflecting the consequence of the decision process. Furthermore, we also found signals in the dorsomedial frontal cortex that resemble the output of a decision comparator, which implicates this region in the computation of the decision itself.acc ͉ action value ͉ reinforcement learning ͉ sma ͉ vmpfc C onsider a goalkeeper trying to stop a soccer ball during a penalty kick. Within a brief amount of time he needs to choose between jumping to the left or right goal posts. Repeated play against the same opponents allows him to learn about their scoring tendencies, which can be used to compute the values of a left and a right jump before making a decision. It is a long-established view in economics, psychology, and computational neuroscience that the brain makes choices among actions by first computing a value for each possible action, and then selecting one of them on the basis of those values (1-3). This raises two fundamental questions in decision neuroscience: (1) where in the brain are the values of different types of actions encoded? and (2) how and where does the brain compare those values to generate a choice?An emerging theme in decision neuroscience is that organisms need to make a number of value-related computations to make even simple choices (4). Consider the case of action-based choice exemplified by the goalkeeper's problem. First, he needs to assign a value to each action under consideration. These signals, known as action values, encode the value of each action before choice and regardless of whether it is subsequently chosen or not, which allows them to serve as inputs into the decision-making process (5-7). Second, these action values are compared to generate a choice.

The left hemisphere learns what is right: Hemispatial reward learning depends on reinforcement learning processes in the contralateral hemisphere

“…Another interpretation of the links between spatial learning biases and hemisphere-specific reinforcement learning mechanisms, as we report here, can be derived from research on multieffector action reinforcement which posits that different actions are learned and associated to the effectors performing them (Daw, 2014;Madlon-Kay et al, 2013;Wunderlich et al, 2009). For example, PEs for actions performed by the right or left hand are predominantly represented in the contralateral ventral striatum (Gershman et al, 2009;Madlon-Kay et al, 2013;Palminteri et al, 2009).…”

“…These findings, together with findings highlighting a crucial role for DA in activational aspects of behaviour Everitt, 1992, 2007), have led researchers to suggest that the impact of motivation on behaviour is mediated by DA (Chakravarthy et al, 2010;Salamone and Correa, 2012;Wise, 2004). Several studies have also shown that the same neural circuitry may also represent the expected value of stimuli presented to one hemifield, but largely restricted to the contralateral hemisphere (Gershman et al, 2009;Palminteri et al, 2009;Wunderlich et al, 2009), suggesting the possibility of unilaterally activating the brain's motivational system. This notion was recently confirmed in a study where unilateral deep brain stimulation of the subthalamic nucleus, a procedure mimicking the impact of DA enhancers, boosted motivational processes within the stimulated hemisphere, as indicated by greater exerted force for the hand controlled by the stimulated hemisphere following increased monetary incentives (Palminteri et al, 2013).…”

“…While incentive motivation can be increased in one hemisphere separately (Palminteri et al, 2013;Schmidt et al, 2010), and the neural representation of reward values for lateralized presented objects are located in the contralateral hemisphere (Gershman et al, 2009;Palminteri et al, 2009;Wunderlich et al, 2009), it remains unclear whether the neural correlates of hemifield-specific incentive motivation are also lateralized. The present results demonstrate that subtle differences in incentive motivation between hemifields, implicating lateralized brain structures during learning (i.e.…”

Economic choices can be made using only stimulus values

“…Note that the prediction was that these chosen value signals would be observed before any action-related information was made available to the subjects, thus making it unlikely that they would be able to process the choice in the motor system. Based on several previous fMRI studies, we expected to see a neuronal representation of the chosen value in ventromedial prefrontal cortex (vmPFC) (12,25,26). If our hypothesis is correct, it will provide unique evidence that the brain can compute choices solely in goods space.…”

“…Although this action based model of making decisions may seem convoluted, it is in fact the predominant view among many decision neuroscientists who have found value signals in areas of the brain known to be involved in representing and planning movements such as lateral parietal and premotor cortices (8)(9)(10)(11). Further evidence for the view that decisions are computed by a comparison between actions comes from the finding of action value signals in several regions of the brain, including the caudate nucleus (5,6), supplementary motor cortex (12), and action-related value signals in lateral intraparietal cortex (9,13). Additionally, the action-based model is sometimes presented as a more general psychological model of behavior because it builds more or less directly on theories of reinforcement learning (RL) and seems to provide a flexible and adaptable unitary model for universal problem solving.…”