Dynamic Flexibility in Striatal-Cortical Circuits Supports Reinforcement Learning

Abstract

Complex learned behaviors must involve the integrated action of distributed brain circuits. While the contributions of individual regions to learning have been extensively investigated, much less is known about how distributed brain networks orchestrate their activity over the course of learning. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time-resolved descriptions of network coordination during reinforcement learning in humans. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex (n=22, 13 females). Moreover, we found that this flexibility in striatal network coupling correlates with participants' learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that episodic learning, measured separately in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal-centered networks provide a mechanism for information integration during reinforcement learning.Learning from the outcomes of actions - referred to as - is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of updating values for actions or cues. Missing from this account, however, is an understanding of how different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal-cortical network dynamics that relate to reinforcement learning behavior.

The energy landscape underpinning module dynamics in the human brain connectome

“…A natural question following the observation of these modules was “What do they do? And how are they recruited as we go through life performing a variety of functions?” To address these questions, dynamic community detection methods were developed and applied to neuroimaging data, revealing the fact that modules reconfigure in support of working memory (Braun et al, 2015, 2016), reinforcement learning (Gerraty et al, 2016), visuo-motor learning (Bassett et al, 2011, 2013b, 2015), and linguistic processing (Chai et al, 2017; Doron et al, 2012a). Module reconfiguration at rest has also been reported as a marker of aging and development (Betzel et al, 2015).…”

Cohesive network reconfiguration accompanies extended training

“…Prior work in multilayer network approaches has largely focused on network flexibility, which is a powerful metric for understanding changes in network communities during both task performance and during the resting state. When estimated across the whole brain, network flexibility is correlated with individual differences in motor skill learning [Bassett et al, ] and in reinforcement learning [Gerraty et al, ], changes within a single subject according to the subject's affect (positive vs. negative; aroused vs. not aroused) and level of fatigue [Betzel et al, ], and is modulated by the NMDA‐receptor antagonist Dextromethorphan suggesting its dependence on glutamate function [Braun et al, ]. When estimated in specific regions of the brain such as the frontal cortex, network flexibility has been shown to increase during high cognitive load in a 2‐back working memory task [Braun et al, ], and correlate with individual differences in cognitive flexibility [Bassett et al, ] and working memory accuracy [Braun et al, ].…”

“…Network flexibility demonstrates low values in healthy controls, intermediate values in siblings of people with schizophrenia, and high values in people with schizophrenia [Braun et al, ]. Finally, individual differences in network flexibility have been shown to correlate with individual differences in performance on tasks requiring executive function including motor learning [Bassett et al, ] and reinforcement learning [Gerraty et al, ].…”

Atypical flexibility in dynamic functional connectivity quantifies the severity in autism spectrum disorder

“…Recent work in the field of network flexibility has centered around the relationship between task-performance or task-sentiment analysis with dynamic reconfiguration of the brain (Park et al, 2017;Telesford et al, 2017) especially in the memory areas (Douw et al, 2015). Intra-subject changes in flexibility have been associated with mood (Betzel et al, 2017), while inter-subject differences have been linked to learning (Bassett et al, 2011), working memory performance (Braun et al, 2015), and reinforcement learning (Gerraty et al, 2018). The metric has also been found to correlate with schizophrenia risk, and is altered by an NMDA-receptor antagonist (Braun et al, 2016).…”

“…Dynamic functional connectivity has been shown to reveal patterns of brain states that occur commonly as well as the transitions between them Preti et al, 2017). They also give an idea about dynamic reconfiguration that occurs during tasks (Bassett et al, 2011;Braun et al, 2016;Gerraty et al, 2018). A common approach used to perform dynamic connectivity analysis is to divide the scan session into overlapping sub-intervals or windows, and calculate a full correlation matrix for each sub-interval.…”