Noradrenergic Regulation of Two-Armed Bandit Performance

Abstract

Recent studies have established that one-trial-back decision policies (Win-Stay/Lose-Shift) and measures of reinforcement learning (RL), e.g. learning rate, can explain how animals perform two-armed bandit tasks. In many published studies, outcomes reverse after one option is selected repeatedly (e.g. 8 selections in a row), and the primary measure of performance is the number of reversals completed. Performance and Win-Stay likelihood are confounded by using recent performance to drive reversals. An alternative design reverses outcomes across options over fixed blocks of trials. We used this blocked design and tested rats in a spatial two-armed bandit task. We analyzed performance using Win-Stay/Lose-Shift (WSLS) metrics and a RL algorithm. We found that WSLS policies remain stable with increasing reward uncertainty, while choice accuracy decreases. Within test sessions, learning rates increased as rats adapted their strategies over the first few reversals but inverse temperature remains stable. We found that muscimol inactivation of medial orbital cortex (mOFC) mediates task performance and negative feedback sensitivity. Finally, we examined the role of the adrenergic system in bandit performance, and found yohimbine (2 mg/kg) dramatically decreased sensitivity to positive feedback, leading to decreases in accuracy and inverse temperature. These effects are partially dependent on a2 adrenergic receptors in OFC. Our findings demonstrate a correspondence between reward schedule, WSLS policies and RL metrics in a task design that is free of the confound between Wins and reversals, and that the noradrenergic influence of mOFC on WSLS policy is dissociable from the regions general role in cognitive flexibility.

Exploration-exploitation mechanisms in recurrent neural networks and human learners in restless bandit problems

“…One caveat of this work is that, although often applied in the context of RL in volatile environments (Domenech et al, 2020;Kovach et al, 2012;Swanson et al, 2020), the comparison between stay and switch trials does not unequivocally map onto the exploitation vs. exploration distinction. For example, stay decisions can be due to greedy choices (choosing the option with the highest expected reward) but also due to perseveration.…”

“…In humans, exploratory choices are associated with increased activity in the frontoparietal network (Beharelle et al, 2015;Chakroun et al, 2020;Daw et al, 2006;Wiehler et al, 2021) and regulated by dopamine and norepinephrine neuromodulatory systems (Chakroun et al, 2020;Cremer et al, 2023;Dubois et al, 2021;McClure et al, 2005;Swanson et al, 2020). Choice predictive signals in prefrontal cortex neural populations are disrupted during exploratory choices, reflecting a potential neural mechanism for random exploration (Ebitz et al, 2018).…”

“…In humans, exploratory choices are associated with increased activity in the frontoparietal network [6,11,13,24] and regulated by dopamine and norepinephrine neuromodulatory systems [11,[25][26][27][28]. Choice predictive signals in prefrontal cortex neural populations are disrupted during exploratory choices, reflecting a potential neural mechanism for random exploration [7].…”