Executive function supports single-shot endowment of value to arbitrary transient goals

Abstract

Recent evidence suggests that executive processes shape reinforcement learning (RL) computations. Here, we extend this idea to the processing of choice outcomes, asking if executive function and RL interact during learning from novel goals. We designed a task where people learned from familiar rewards or abstract instructed goals. We hypothesized that learning from these goals would produce reliable responses in canonical reward circuits, and would do so by leveraging executive function. Behavioral results pointed to qualitatively similar learning processes when subjects learned from achieving goals versus familiar rewards. Goal learning was robustly and selectively correlated with performance on an independent executive function task. Neuroimaging revealed comparable appetitive responses and computational signatures in reinforcement learning circuits for both goal-based and familiar learning contexts. During goal learning, we observed enhanced correlations between prefrontal cortex and canonical reward-sensitive regions, including hippocampus, striatum, and the midbrain. These findings demonstrate that attaining novel goals produces reliable reward signals in dopaminergic circuits. We propose that learning from goal-directed behavior is mediated by top-down input that primes the reward system to endow value to cues signaling goal attainment.

How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective

“…For example, WM may maintain reward information itself (deviating from traditional theories in which reward information is only stored by RL processes): in the PFC-BG model developed by (Zhao et al, 2018), dopaminergic signals update both basal ganglia and prefrontal cortex, where reward information is encoded and updated in WM. Similarly, recent imaging work shows that WM helps transform novel goal stimuli into a signal the brain interprets as rewards for learning (McDougle et al, 2021). WM may assist RL by representing more abstract task-relevant information, to allow for generalization across tasks (Williams & Phillips, 2020), or by effectively lowering the set of states or actions RL operates over by filtering out irrelevant state spaces (Rmus et al, 2021).…”