Orbitofrontal cortex and learning predictions of state transitions.

Chan, Stephanie C. Y.; Lopatina, N. G.; Schoenbaum, Geoffrey; Niv, Yaron

doi:10.1037/bne0000461

“…Within the presented framework, symbolic reinforcers have been recast as dimensions that drive the state value. Past work involving choice tasks and the state value has suggested the existence of a ventral circuit for the representation of the state value ( Gläscher et al, 2010 ; Averbeck and Murray, 2020 ) and state transitions ( Belova et al, 2008 ; Chan et al, 2021 ; Kalmbach et al, 2022 ). It has been hypothesized that distinct ventral and dorsal networks define behavioral goals and orchestrate actions to achieve goals, respectively ( Everitt et al, 1999 ; Cardinal et al, 2002 ; Averbeck and Costa, 2017 ; Averbeck and Murray, 2020 ).…”

Section: Discussionmentioning

confidence: 99%

Computational Mechanisms Underlying Motivation to Earn Symbolic Reinforcers

Burk,

Taswell,

Tang

et al. 2024

J. Neurosci.

1

0

View full text Add to dashboard Cite

Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that are later exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are ubiquitous in our daily lives, widely used in laboratory tasks because they can be motivating, mechanisms by which they become motivating are less understood. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to then correlate with the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys, three male and two female). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.Significance statementSymbolic reinforcers, like money and points, play a critical role in our lives. However, we lack a mechanistic understanding of how symbolic reinforcement can be related to fluctuations in motivation. We investigated the effect of symbolic reinforcers on behaviors related to motivation during a token reinforcement learning task, using a novel reinforcement learning model and data from five monkeys. We designed a state-based model that can capture reward-predicting features and produce state values at sub-trial resolution. Our findings suggest that the value of a task state can affect willingness to initiate a trial, speed to choose, and persistence to complete a trial. Our model makes testable predictions for within trial fluctuations of neural activity related to values of task states.

show abstract

“…Within the presented framework, symbolic reinforcers have been recast as dimensions that drive state value. Past work involving choice tasks and state value have suggested the existence of a ventral circuit for the representation of state value (Gläscher et al, 2010; Averbeck and Murray, 2020) and state transitions (Belova, Paton and Salzman, 2008; Chan et al, 2021; Kalmbach et al, 2022). It has been hypothesized that distinct ventral and dorsal networks define behavioral goals and orchestrate actions to achieve goals, respectively (Everitt et al, 1999; Cardinal et al, 2002; Averbeck and Costa, 2017; Averbeck and Murray, 2020).…”

Section: Discussionmentioning

confidence: 99%

Computational mechanisms underlying motivation to earn symbolic reinforcers

Burk,

Taswell,

Tang

et al. 2023

Preprint

1

0

View full text Add to dashboard Cite

Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that can later be exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are motivating, little is understood about the neural or computational mechanisms underlying the motivation to earn them. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to capture the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.

show abstract

“…Other regions connected with the BLA, including the orbitofrontal cortex (OFC), have a key role in outcome- and state-based decision making (Wallis 2007; Schuck et al 2016). In particular, OFC is important for discriminating state transitions, which is necessary for making appropriate decisions in dynamic environments (Bradfield et al 2015; Stalnaker et al 2021; Chan et al 2021). Drug exposure may impair this process, resulting in persistent drug-seeking regardless of availability state or in the face of exaggerated costs (Lucantonio et al 2012).…”

Section: Discussionmentioning

confidence: 99%

Hierarchical cue control of drug seeking in the face of cost

Collins

¹

,

Bornhoft

²

,

Wolff

³

et al. 2022

Preprint

1

0

View full text Add to dashboard Cite

Addiction is characterized by intermittent drug seeking despite rising costs. This behavior is heavily influenced by environmental stimuli that signal drug availability and reinforce seeking. We aimed to establish the relationship between three key aspects of human drug use in rats: the intermittent, binge nature of drug intake, the motivational conflict of drug seeking in the face of escalating negative costs, and the ability of different drug cues to interact to modulate relapse. Rats were trained to self administer cocaine on an intermittent access schedule, where brief drug availability states were signaled by a shift in the ambient lighting of the environment and cocaine delivery was signaled by a separate proximal cue. Rats then went through a conflict procedure, where foot shock intensity associated with cocaine seeking was escalated until intake was suppressed. We completed relapse tests where the drug delivery cue was non contingently presented alone, or in the context of dynamic drug availability state transitions. Intermittent access spurred psychomotor sensitization and binge-like cocaine intake. The intensity of binge-like drug taking during training was predictive of later drug seeking despite escalating costs. In relapse tests, the ability of a proximal drug cue to trigger relapse was gated by the presence of a global cue signaling drug-availability state transitions. Our results suggest that the pattern of drug intake plays a role in many features of addiction, including modifying an individual's willingness to endure high costs associated with drug seeking. Further, our results indicate that drug-related sensory information can be hierarchically organized to exert a dynamic modulating influence on drug-seeking motivation.

show abstract

Orbitofrontal cortex and learning predictions of state transitions.

Cited by 8 publications

References 68 publications

Computational Mechanisms Underlying Motivation to Earn Symbolic Reinforcers

Computational Mechanisms Underlying Motivation to Earn Symbolic Reinforcers

Computational mechanisms underlying motivation to earn symbolic reinforcers

Hierarchical cue control of drug seeking in the face of cost

Contact Info

Product

Resources

About