Cortical substrates for exploratory decisions in humans

Abstract

Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'explorationexploitation' dilemma1, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning2 (RL) theory, indicates that a dopaminergic3,4, striatal5-9 and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical1 and ethological10 perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.Exploration is a computationally refined capacity, demanding careful regulation. Two possibilities for this regulation arise. On the one hand, we might expect the involvement of cognitive, prefrontal control systems11 that can supervene12 over simpler dopaminergic/ striatal habitual mechanisms. On the other hand, theoretical work on optimal exploration1,13 indicates a more unified architecture, according to which actions can be assessed with the use of a metric that integrates both primary reward and the informational value of exploration, even in simple, habitual decision systems.We studied patterns of behaviour and brain activity in 14 healthy subjects while they performed a 'four-armed bandit' task involving repeated choices between four slot machines ( Fig. 1; see Supplementary Methods). The slots paid off points (to be exchanged for money) noisily around four different means. Unlike standard slots, the mean payoffs changed randomly and independently from trial to trial, with subjects finding information about the current worth of a slot only through sampling it actively. This feature of the experimental design,...

Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling.

“…where s(t) was the agent's current state; and ␤ was a parameter that balanced exploration and exploitation (Daw, O'Doherty, Dayan, Seymour, & Dolan, 2006;Doya, 2002). High ␤ drove the agent to be more likely to select the high-value action, whereas low ␤ made the agent more likely to select actions more randomly.…”

“…Humans and animals can show different balances between exploration and exploitation (Daw, O'Doherty, et al, 2006;Doya, 2000b). Animals faced with extinction conditions often increase their responding to the stimulus and also can show increased exploration (Domjan, 1998;Ferster & Skinner, 1957).…”

Risky business: the neuroeconomics of decision making under uncertainty

“…This tension between seeking new information and choosing the best option, given what is already known, is called the 'explore-exploit' dilemma. In a study investigating the potential neural basis for such trade-offs 66 , subjects chose between four virtual slot machines, each with a different, unknown, and changing payoff structure. When subjects received large rewards from their chosen machine, there was increased activation in the ventromedial prefrontal cortex, with deviations from an expected reward level represented in the amplitude of ventral striatal activation.…”

Seeing is believing: Trustworthiness as a dynamic belief

“…This means that while the models were fit to each participant's individual behavioral data, the error in the parameter estimation was pooled across subjects. This procedure has been previously utilized when individual parameter estimates are not stable as a result of a small number of trials and collinearity between parameters (Daw, O'Doherty, Dayan, Seymour, & Dolan, 2006). While we observe a similar hierarchy of model fits when an individual parameter is fit for each participant, we report the more stable group fits.…”