The Role of Executive Function in Shaping Reinforcement Learning

Rmus, Milena; McDougle, Samuel D.; Collins, Anne

doi:10.31234/osf.io/9cvw3

Cited by 14 publications

(22 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, RL as a computational model of cognition typically assumes a given action space defined by the modeler, which provides the relevant dimensions of the choice space (i.e. either the yogurt color or the cup position) -there is no ambiguity in what choices are, and the nature of the choice space does not matter (Rmus, McDougle, & Collins, 2020). As such, RL experiments in psychology tend to not consider the type of choices (a single motor-action such as pressing a key with the index finger; (A. G. E. Collins, Ciullo, Frank, & Badre, 2017;Tai, Lee, Benavidez, Bonci, & Wilbrecht, 2012), or the selection of a goal stimulus (Foerde & Shohamy, 2011;Daw, Gershman, Seymour, Dayan, & Dolan, 2011;Frank, Moustafa, Haughey, Curran, & Hutchison, 2007)) as important, and researchers use the same models and generalize findings across choice types.…”

Section: Introductionmentioning

confidence: 99%

Plucking a string or playing a G? Choice type impacts human reinforcement learning

Rmus

Zou

Collins

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values which change incrementally. RL models consider any response type indiscriminately, ranging from less abstract choices (e.g. pressing a key with the index finger), to more abstract choices that can be executed in a number of ways (e.g. getting dinner at the restaurant). But does the learning process vary as a function of how abstract the choices are? In Experiment 1, we show that choice abstraction impacts learning: participants were slower and less accurate in learning to select a more abstract choice. Using computational modeling, we show that two mechanisms contribute to this. First, the values of motor actions interfered with the values of more abstract responses, resulting in more incorrect choices; second, information integration for relevant abstract choices was slower. In Experiment 2, we replicate the findings from Experiment 1, and further extend the results by investigating whether slowed learning is attributable to working memory (WM) or RL contributions. We find that the impairment in more abstract/flexible choices is driven primarily by a weaker contribution of WM. We conclude that defining a more abstract choice space used by multiple learning systems recruits limited executive resources, limiting how much such processes then contribute to fast learning.

show abstract

Section: Introductionmentioning

confidence: 99%

Plucking a string or playing a G? Choice type impacts human reinforcement learning

Rmus

Zou

Collins

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our study provides quantitative evidence that a pure reinforcement learning modeling approach does not capture the cognitive processes needed to solve feature-based learning. By formalizing 20 the subcomponent processes needed to augment standard RL modeling we provide strong empirical evidence for the recently proposed 'EF-RL' framework that describes how executive functions (EF) augment RL mechanism during cognitive tasks (Rmus et al, 2020). The framework asserts that RL mechanisms are central for learning a policy to address task challenges, but that attention-, action-, and higher-order expectations are integral for shaping these policies (Rmus et al, 2020).…”

Section: Resultsmentioning

confidence: 99%

“…By formalizing 20 the subcomponent processes needed to augment standard RL modeling we provide strong empirical evidence for the recently proposed 'EF-RL' framework that describes how executive functions (EF) augment RL mechanism during cognitive tasks (Rmus et al, 2020). The framework asserts that RL mechanisms are central for learning a policy to address task challenges, but that attention-, action-, and higher-order expectations are integral for shaping these policies (Rmus et al, 2020). In our study these 'EF' functions included working memory, adaptive exploration, and an 'attention' mechanism for decaying nonchosen values.…”

Section: Resultsmentioning

confidence: 99%

Learning at variable attentional load requires cooperation between working memory, meta-learning and attention-augmented reinforcement learning

Womelsdorf

Watson

Tiesinga³

2020

Preprint

View full text Add to dashboard Cite

Flexible learning of changing reward contingencies can be realized with different strategies. A fast learning strategy involves forming a working memory of rewarding experiences with objects to improve future choices. A slower learning strategy uses prediction errors to gradually update value expectations to improve choices. How the fast and slow strategies work together in scenarios with real-world stimulus complexity is not well known. Here, we disentangle their relative contributions in rhesus monkeys while they learned the relevance of object features at variable attentional load. We found that learning behavior across six subjects is consistently best predicted with a model combining (i) fast working memory (ii) slower reinforcement learning from positive and negative prediction errors, as well as (iii) selective suppression of non-chosen features values, and (iv) meta-learning based adjustment of exploration rates given a memory trace of recent errors. These mechanisms cooperate differently at low and high attentional loads. While working memory was essential for efficient learning at lower attentional loads, learning from negative outcomes and metalearning were essential for efficient learning at higher attentional loads. Together, these findings highlight that a canonical set of distinct learning mechanisms cooperate to optimize flexible learning when adjusting to environments with real-world attentional demands.

show abstract

“…It is often referred to as model-free learning, in contrast to computationally taxing but often more efficient model-based learning, where the agent uses a model of the world to simulate possible paths and downstream outcomes of a decision (Daw et al, 2011), a process thought to depend on the prefrontal cortex (Otto, Gershman, et al, 2013;Smittenaar et al, 2013). Some of the more sophisticated aspects of human decision-making previously unexplained by model-free RL -generalizing across contexts, judging what information is currently relevant, breaking down big goals into subgoals -depend on interactions of canonical RL with cognitive control processes subserved by fronto-parietal, cingulo-opercular networks (Botvinick, 2012; J. F. Cavanagh & Frank, 2014;Collins et al, 2017;Otto, Gershman, et al, 2013;Rmus et al, 2020). On the other hand, cognitive maps that support learning in physical and abstract spaces depend on the hippocampus (Dombrovski et al, 2020;Mattar & Daw, 2018;Miller et al, 2017;Vikbladh et al, 2019).…”

Section: Reinforcement Learning (Sidebar 2)mentioning

confidence: 99%

Search for solutions, learning, simulation, and choice processes in suicidal behavior

Dombrovski¹,

Hallquist²

2021

Preprint

View full text Add to dashboard Cite

Suicide may be viewed as an unfortunate outcome of failures in decision processes. Such failures occur when the demands of a crisis exceed a person’s capacity to (i) search for options, (ii) learn and simulate possible futures, and (iii) make advantageous value-based choices. Can individual-level decision deficits and biases drive the progression of the suicidal crisis? Our overview of the evidence on this question is informed by clinical theory and grounded in reinforcement learning and behavioral economics. Cohort and case-control studies provide strong evidence that limited cognitive capacity and particularly impaired cognitive control are associated with suicidal behavior, imposing cognitive constraints on decision-making. We conceptualize suicidal ideation as an element of impoverished consideration sets resulting from a search for solutions under cognitive constraints and mood-congruent Pavlovian influences, a view supported by mostly indirect evidence. More compelling is the evidence of impaired learning in people with a history of suicidal behavior. We speculate that an inability to simulate alternative futures using one’s model of the world may undermine alternative solutions in a suicidal crisis. The hypothesis supported by the strongest evidence is that the selection of suicide over alternatives is facilitated by a choice process undermined by randomness. Case-control studies using gambling tasks, armed bandits and delay discounting support this claim. Future experimental studies will need to uncover real-time dynamics of choice processes in suicidal people. In summary, the decision process framework sheds light on neurocognitive mechanisms that facilitate the progression of the suicidal crisis.

show abstract

The Role of Executive Function in Shaping Reinforcement Learning

Cited by 14 publications

References 25 publications

Plucking a string or playing a G? Choice type impacts human reinforcement learning

Plucking a string or playing a G? Choice type impacts human reinforcement learning

Learning at variable attentional load requires cooperation between working memory, meta-learning and attention-augmented reinforcement learning

Search for solutions, learning, simulation, and choice processes in suicidal behavior

Contact Info

Product

Resources

About