Dynamic flexibility in striatal-cortical circuits supports reinforcement learning

Abstract

Complex learned behaviors must involve the integrated action of distributed brain circuits. While the contributions of individual regions to learning have been extensively investigated, understanding how distributed brain networks orchestrate their activity over the course of learning remains elusive. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time--resolved descriptions of network coordination during reinforcement learning. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex. Moreover, we found that flexibility in striatal network dynamics correlates with participants' learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that not all forms of learning relate to this circuit: episodic memory, measured in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal--centered networks provide a mechanism for information integration during reinforcement learning. Significance Statement Learning from the outcomes of actions ----referred to as reinforcement learning ----is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of the updating of values for actions or sensory stimuli. Missing from this account, however, is a description of the manner in which different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal--cortical network dynamics that relate to reinforcement learning behavior.. CC-BY-NC-ND 4.0 International license peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/094383 doi: bioRxiv preprint first posted online Dec. 15, 2016; 1 Introduction Learning from reinforcement is central to adaptive behavior and requires continuous and dynamic integration of sensory, motor, and reward information over time. Major progress has been made in understanding how individual brain regions support reinforcement learning. However, remarkably little is known about how these brain regions interact during learning, how their interactions change over time, and how these dynamic circuit--level changes relate to successful learning. In a typical reinforcement learning task, participants use reinforcement over hundreds of trials to associate cues or actions with their most probable outcome (e.g. (1--4)). Computationally, this is captured by so--called "model--free" reinforcement learning algorithms, a class of models that provide a quantitative and mechanistic framework for describing behavior (1, 5, 6). These models have also been successful in accounting for neuronal signals underlying learning b...

Dynamic reconfiguration of functional subgraphs after musical training in young adults

“…Greater switching behaviour of the subgraph illustrates increasing flexibility of the interaction between functional systems after musical training. Noteworthy, the flexible interaction has been shown between functional systems during executive cognition [4], reinforcement learning [39], and musical training [25]. In the current study, flexible interaction was found between the primary functional systems, as well as the higherorder cognitive related systems-cingulo-opercular task control and salience systems.…”

Functional brain network reconfiguration during learning in a dynamic environment

“…(c) Whole-brain thresholded activation maps for CPP and RU from McGuire et. al(2014) and whole-brain maps for edge strength of subgraph 4 in the current study. Optimal parameters for nonnegative matrix factorization.…”

“…(b) Relationship between the activation for RU and the edge strength of subgraph 4. We observed a significant positive correlation between the z statistic for RU from McGuire et al(2014) and the edge strength across nodes in subgraph 4. The red line represents the regression line and the shaded area represents the 95% confidence interval.…”

“…(a) Relationship between the activation for CPP and the edge strength of subgraph 4. We calculated the Pearson correlation coefficient between the z-statistic for CPP from McGuire et al(2014) and the edge strength across nodes in subgraph 4. Each data point represents an ROI.…”

Cohesive network reconfiguration accompanies extended training

“…Network flexibility demonstrates low values in healthy controls, intermediate values in siblings of people with schizophrenia, and high values in people with schizophrenia [Braun et al, 2016]. Finally, individual differences in network flexibility have been shown to correlate with individual differences in performance on tasks requiring executive function including motor learning [Bassett et al, 2011] and reinforcement learning [Gerraty et al, 2016]. Yet, despite the success of dynamic network methods generally as well as the neurophysiological relevance of network flexibility for executive function specifically, much work still remains to devise metrics of reconfiguration that best capture neurophysiological processes.…”