Transcranial Stimulation over Frontopolar Cortex Elucidates the Choice Attributes and Neural Mechanisms Used to Resolve Exploration–Exploitation Trade-Offs

Abstract

Optimal behavior requires striking a balance between exploiting tried-and-true options or exploring new possibilities. Neuroimaging studies have identified different brain regions in humans where neural activity is correlated with exploratory or exploitative behavior, but it is unclear whether this activity directly implements these choices or simply reflects a byproduct of the behavior. Moreover, it remains unknown whether arbitrating between exploration and exploitation can be influenced with exogenous methods, such as brain stimulation. In our study, we addressed these questions by selectively upregulating and downregulating neuronal excitability with anodal or cathodal transcranial direct current stimulation over right frontopolar cortex during a reward-learning task. This caused participants to make slower, more exploratory or faster, more exploitative decisions, respectively. Bayesian computational modeling revealed that stimulation affected how much participants took both expected and obtained rewards into account when choosing to exploit or explore: Cathodal stimulation resulted in an increased focus on the option expected to yield the highest payout, whereas anodal stimulation led to choices that were less influenced by anticipated payoff magnitudes and were more driven by recent negative reward prediction errors. These findings suggest that exploration is triggered by a neural mechanism that is sensitive to prior less-than-expected choice outcomes and thus pushes people to seek out alternative courses of action. Together, our findings establish a parsimonious neurobiological mechanism that causes exploration and exploitation, and they provide new insights into the choice features used by this mechanism to direct decision-making.

Attenuated directed exploration during reinforcement learning in gambling disorder

“…While this network was initially characterized in the context of random exploration (Daw et al, 2006), recent evidence points towards a role of the fronto-polar cortex specifically in directed exploration (Badre et al, 2012;Boorman, Behrens, Woolrich, & Rushworth, 2009;Boorman, Behrens, & Rushworth, 2011;Zajkowski, Kossut, & Wilson, 2017). This is also supported by causal manipulations using transcranial magnetic stimulation (Zajkowski et al, 2017) and transcranial direct current stimulation (Raja Beharelle et al, 2015).…”

“…Exploration is supported by a bilateral fronto-parietal network including intra-parietal sulcus and frontopolar cortex. (Badre, Doll, Long, & Frank, 2012;Daw et al, 2006;Raja Beharelle, Polania, Hare, & Ruff, 2015). While this network was initially characterized in the context of random exploration (Daw et al, 2006), recent evidence points towards a role of the fronto-polar cortex specifically in directed exploration (Badre et al, 2012;Boorman, Behrens, Woolrich, & Rushworth, 2009;Boorman, Behrens, & Rushworth, 2011;Zajkowski, Kossut, & Wilson, 2017).…”

The modulating effects of brain stimulation on emotion regulation and decision-making

“…The results indicate that the right DLPFC is involved with the sunk-cost effect. In an exploration (trying something new)-exploitation (sticking with a proven strategy) trade-offs study using tDCS [61], separated groups received anodal, cathodal, or sham tDCS over the right frontopolar cortex (FPC). Cathodal stimulation increased choices of the highest rewards, but anodal stimulation induced choices affected less by anticipated rewards and more by recent negative reward prediction errors, indicating that the right FPC is involved in controlling both exploration and exploitation.…”

“…In summary, tDCS influences and modulates risk-taking behaviors [5,12,18,19,47,82,83], choice modulation [46], delayed discounting [30], maladaptive decisionmaking [81], probabilistic guessing [29], moral judgment [37], sunk-cost effect [4], exploration-exploitation trade-offs [61], decision-making and cognitive impulse control [42], perception of space and time [79], dual-task performance [22], model-based learning [70], addiction [6,17,26,60,80], food craving [23,35], and perceptual decision-making [34].…”

Frontostriatal anatomical connections predict age- and difficulty-related differences in reinforcement learning

“…Condition differences may depend on how well an individual adapts to such an environment, or the willingness to adjust the learning rate. The ACC has been related to learning rate adjustments with changing circumstances (Behrens et al, 2007), and aPFC is involved in the arbitration between exploration and exploitation (Boorman et al, , 2009Beharelle et al, 2015). Thus, age-related changes in learning rates, their M A N U S C R I P T…”