fMRI and EEG Predictors of Dynamic Decision Parameters during Human Reinforcement Learning

Frank, Michael J.; Gagné, Christopher; Nyhus, Erika; Masters, Sean; Wiecki, Thomas V.; Cavanagh, James F.; Badre, David

doi:10.1523/jneurosci.2036-14.2015

Cited by 225 publications

(310 citation statements)

References 44 publications

Supporting

Mentioning

265

Contrasting

Order By: Relevance

“…In line with a number of these accounts, we recently proposed that dACC integrates controlrelevant values (including factors such as reward, conflict, and error likelihood) to make adjustments to candidate control signals (Shenhav, Botvinick, & Cohen, 2013). In this setting, one of multiple potentially relevant control signals is the decision threshold for the current and future trials, adjustments of which have been found to be triggered by current trial conflict (i.e., difficulty) and mediated by dACC and surrounding regions (Cavanagh & Frank, 2014;Cavanagh et al, 2011;Danielmeier, Eichele, Forstmann, Tittgemeyer, & Ullsperger, 2011;Frank et al, 2015;Kerns et al, 2004). Our findings also do not rule out the possibility that dACC activity will in other instances track the likelihood of switching rather than sticking with one's current strategy-as has been observed in numerous studies of default override (see Shenhav et al, 2013)-over and above signals related to choice difficulty.…”

Section: Discussionmentioning

confidence: 99%

Dorsal anterior cingulate and ventromedial prefrontal cortex have inverse roles in both foraging and economic choice

Shenhav

Straccia

Botvinick

et al. 2016

Cogn Affect Behav Neurosci

View full text Add to dashboard Cite

Recent research has highlighted a distinction between sequential foraging choices and traditional economic choices between simultaneously presented options. This was partly motivated by observations in Kolling, Behrens, Mars, and Rushworth, Science, 336(6077), 95-98 (2012) (hereafter, KBMR) that these choice types are subserved by different circuits, with dorsal anterior cingulate (dACC) preferentially involved in foraging and ventromedial prefrontal cortex (vmPFC) preferentially involved in economic choice. To support this account, KBMR used fMRI to scan human subjects making either a foraging choice (between exploiting a current offer or swapping for potentially better rewards) or an economic choice (between two reward-probability pairs). This study found that dACC better tracked values pertaining to foraging, whereas vmPFC better tracked values pertaining to economic choice. We recently showed that dACC's role in these foraging choices is better described by the difficulty of choosing than by foraging value, when correcting for choice biases and testing a sufficiently broad set of foraging values (Shenhav, Straccia, Cohen, & Botvinick Nature Neuroscience, 17(9), 1249-1254. Here, we extend these findings in 3 ways. First, we replicate our original finding with a larger sample and a task modified to address remaining methodological gaps between our previous experiments and that of KBMR. Second, we show that dACC activity is best accounted for by choice difficulty alone (rather than in combination with foraging value) during both foraging and economic choices. Third, we show that patterns of vmPFC activity, inverted relative to dACC, also suggest a common function across both choice types. Overall, we conclude that both regions are similarly engaged by foraging-like and economic choice.

show abstract

Section: Discussionmentioning

confidence: 99%

Dorsal anterior cingulate and ventromedial prefrontal cortex have inverse roles in both foraging and economic choice

Shenhav

Straccia

Botvinick

et al. 2016

Cogn Affect Behav Neurosci

View full text Add to dashboard Cite

show abstract

“…Second, the Bayesian hierarchical framework has been shown to be especially effective when the number of observations is low [28]. Third and more importantly, this framework supports the use of other variables as regressors of the model parameters to assess relations of the model parameters with other physiological or behavioral data [29][30][31][32]. This property of the HDDM allowed us to establish the link between the results of the brain-behavior coupling analysis and the model parameters.…”

Section: Hierarchical Drift Diffusion Modelling Ofmentioning

confidence: 72%

Correlations of Neural Activity with Behavioral Kinematics during Active Tactile Decision Making

Delis

Dmochowski

Sajda

et al. 2018

Proceedings of the 10th Hellenic Conference on Artificial Intelligence

View full text Add to dashboard Cite

Most real-world decisions rely on active sensing, a dynamic process for directing our sensors (e.g. eyes or fingers) across a stimulus in order to reduce uncertainty and maximize information gain. Though ecologically pervasive, relatively limited work has focused on identifying neural correlates of the active sensing process. In tactile perception, we often make decisions about an object or surface by actively exploring its shape and texture. Here we investigate the neural mechanisms of active tactile sensing by simultaneously measuring electroencephalography (EEG) and finger kinematics while subjects interrogated a haptic surface to make perceptual judgements. We hypothesized that one's sensorimotor behavior provides a view into the cognitive processes leading to decision formation, and the neural correlates of these processes would be detectable by relating kinematics to neural activity. Using an adaptation of canonical correlation analysis (CCA), we regressed the EEG onto kinematics and found three distinct, task-related EEG components that localized to rightlateralized occipital cortex (LOC), middle frontal gyrus (MFG), and supplementary motor area (SMA), respectively. To probe the functional role of these components, we fit their single-trial activity to behavior using a hierarchical drift diffusion model (HDDM), revealing that the LOC modulated the encoding of the tactile stimulus whereas the MFG predicted the rate of information integration towards a choice. This study provides direct evidence that, how we explore the stimulus yields insight into how our brain

show abstract

“…In addition to this rapid cortically-mediated form of adaptation, evidence suggests that strategic adjustments in decision threshold are achieved via activity-dependent plasticity in the connections between STN and GPe [24,58] . In the current study, adaptive changes in the boundary height accounted for the observed post-error slowing in responses following failed stop trials, motivated by neuroimaging and electrophysiological evidence STN-mediated slowing of responses [9,17,26] . For simplicity, boundary adaption was restricted to being unidirectional -increasing after a stop-error and decaying back to, but never below, its original value.…”

Section: Drift-rate: Dopaminergic Modulation Of Corticostriatal Pathwaysmentioning

confidence: 96%

“…One reason for this is that computational models of learning [1,2] and control [3][4][5] have historically emerged from disparate lines of empirical research (see [6,7] for exceptions), adding difficulty to the already challenging task of inferring cognitive phenomena from gross behavioral measures. Recently, however, insights from cognitive and computational neuroscience have begun to shed light on the interaction of cognitive processes in neural circuits, providing additional empirical anchors for grounding theoretical assumptions [8][9][10][11] .…”

Section: Introductionmentioning

confidence: 99%

Errors in action timing and inhibition facilitate learning by tuning distinct mechanisms in the underlying decision process

Dunovan¹,

Verstynen²

2017

Preprint

View full text Add to dashboard Cite

Goal-directed behavior requires integrating action selection processes with learning systems that adapt control using environmental feedback. These functions intersect in the basal ganglia (BG), which has at least two targets of plasticity: a dopaminergic modulation of striatal pathways and cortical modulation of the subthalamic nucleus (STN). Dual learning mechanisms suggests that feedback signals have a multifaceted impact on BG-dependent decisions. Using a hybrid of accumulation-to-bound decision models and reinforcement learning, we modeled the performance of humans in a stop-signal task where participants (N=75) learned the prior distribution of the timing of a stop signal through trial-and-error feedback. Changes in the drift-rate of the action execution process were driven by errors in action timing, whereas adaptation in the boundary height served to increase caution following failed stops. These findings highlight two interactive learning mechanisms for adapting the control of goal-directed actions based on dissociable dimensions of feedback error. . CC-BY4.0 International license peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/153676 doi: bioRxiv preprint first posted online Jun. 22, 2017; Author SummaryMany complex behavioral goals rely on one's ability to regulate the timing of action execution while also maintaining enough control to cancel actions in response to "Stop" cues in the environment. Here we examined how these two fundamental components of behavior become tuned to the control demands of the environment by combining principles of reinforcement learning with accumulator models of decision making. The synthesis of these two theoretical frameworks is motivated by previous work showing that reinforcement learning and control rely on overlapping circuitry in the basal ganglia. Leveraging knowledge about the interaction of learning and control signals in this network, we formulated a computational model in which performance feedback is used to modulate key mechanisms of the decision process to facilitate goal acquisition. Model-based analysis of behavioral data collected on an adaptive stop-signal task revealed two critical learning mechanisms: one that adjusts the accumulation rate of the "Go" signal to errors in action timing and another that exercises caution by raising the height of the execution boundary after a failed Stop trial. We show how these independent learning mechanisms interact over the course of learning, shedding light on the behavioral effects plasticity in different pathways of the basal ganglia.

show abstract

fMRI and EEG Predictors of Dynamic Decision Parameters during Human Reinforcement Learning

Cited by 225 publications

References 44 publications

Dorsal anterior cingulate and ventromedial prefrontal cortex have inverse roles in both foraging and economic choice

Dorsal anterior cingulate and ventromedial prefrontal cortex have inverse roles in both foraging and economic choice

Correlations of Neural Activity with Behavioral Kinematics during Active Tactile Decision Making

Errors in action timing and inhibition facilitate learning by tuning distinct mechanisms in the underlying decision process

Contact Info

Product

Resources

About