Dissociable Neural Correlates of Uncertainty Underlie Different Exploration Strategies

Abstract

<jats:title>Summary</jats:title><jats:p>Most real-world decisions involve a delicate balance between exploring unfamiliar alternatives and committing to the best known option. Uncertainty lies at the core of this “explore-exploit” dilemma, for if all options were perfectly known, there would be no need to explore. Yet despite the prominent role of uncertainty-guided exploration in decision making, evidence for its neural implementation is still sparse. We investigated this question with model-based fMRI (n = 31) using a two-armed bandit task that independently manipulates two forms of uncertainty underlying different exploration strategies. The relative uncertainty between the two options was correlated with BOLD activity in right rostrolateral prefrontal cortex and drove directed exploration, a strategy that adds an uncertainty bonus to each option. The total uncertainty across the two options was correlated with activity in left dorsolateral prefrontal cortex and drove random exploration, a strategy that increases choice stochasticity in proportion to total uncertainty. The subjective estimates of uncertainty from both regions were predictive of subject choices. These results are consistent with a hybrid computational architecture in which different uncertainty computations are performed separately and then combined by downstream decision circuits to compute choice.</jats:p>

A neural and behavioral tradeoff underlies exploratory decisions in normative anxiety

“…Methods, Equations 1,2). We included three additional factors: First, because recent evidence suggests that exploration is aimed at reducing the outcome-variability associated with different options (Badre et al, 2012;Frank et al, 2009;Tomov et al, 2020), we included an outcome-variability factor which was updated by a prediction-error scaled by a learning-rate (Equations 6-7). Second, an additional exploratory strategy that allows detecting changes in the environment is randomly switching machines between trials, and we therefore included a random-switching binary factor (Equation 9).…”

“…Exploration was shown to increase with environmental volatility (Knox et al, 2012), and evidence suggest that exploratory decisions are directed towards options with large outcome variabilities (Badre et al, 2012;Cavanagh et al, 2012;Kayser et al, 2015;Tomov et al, 2020). Imaging studies in humans have shown that exploratory decisions engage the anterior insulae (aINS) as well as prefrontal regions involved in executive control processes, such as the frontopolar cortex (FPC) and the dorsal anterior cingulate cortex (dACC) (Badre et al, 2012;Blanchard and Gershman, 2018;Cavanagh et al, 2012;Daw et al, 2006;Kayser et al, 2015;Laureiro-Martinez et al, 2015;Tomov et al, 2020).…”

“…Exploration was shown to increase with environmental volatility (Knox et al, 2012), and evidence suggest that exploratory decisions are directed towards options with large outcome variabilities (Badre et al, 2012;Cavanagh et al, 2012;Kayser et al, 2015;Tomov et al, 2020). Imaging studies in humans have shown that exploratory decisions engage the anterior insulae (aINS) as well as prefrontal regions involved in executive control processes, such as the frontopolar cortex (FPC) and the dorsal anterior cingulate cortex (dACC) (Badre et al, 2012;Blanchard and Gershman, 2018;Cavanagh et al, 2012;Daw et al, 2006;Kayser et al, 2015;Laureiro-Martinez et al, 2015;Tomov et al, 2020). Interestingly, anxious individuals find uncertainty more aversive and are also more intolerant thereof (Buhr and Dugas, 2009;Charpentier et al, 2017;Grupe and Nitschke, 2013;Hartley and Phelps, 2012;Maner et al, 2007;Maner and Schmidt, 2006), and recent studies have shown that various aspects of uncertainty correlate with a hyper-responsive aINS in anxiety (Grupe and Nitschke, 2013;Paulus and Stein, 2006).…”

“…On one hand, anxiety could decrease exploration because it requires selecting options with uncertain outcomes; yet on the other hand, anxiety could increase exploration by reducing uncertainty in the environment. Previous studies have examined the relationship between uncertainty and anxiety (Grupe and Nitschke, 2013) and separately examined the relationship between uncertainty and exploration (Badre et al, 2012;Cavanagh et al, 2012;Kayser et al, 2015;Tomov et al, 2020). However, little is known about the direct relationship between anxiety and exploratory decision-making, the cognitive factors that contribute to it, and the underlying brain mechanisms.…”

Exploration-related strategy mediates negative coupling between decision-making performance and psychiatric symptoms

“…Furthermore, in environments with changing volatility, different computations underlying decision-making and learning might play an essential role 58,59 . Another caveat is that, while we relate choice stochasticity and preference for previously unchosen options to random and uncertainty-driven exploration strategies respectively, more sophisticated experimental tasks are required for in-depth examinations of the nature of the exploration strategies 8,10,12,60 . Furthermore, while we have shown the correlation between one dimension of psychiatric symptoms and particular computational processes in decision-making, we have left open the question as to how these two are causally related.…”