Dissociable Neural Systems Support the Learning and Transfer of Hierarchical Control Structure

Abstract

Humans can draw insight from previous experiences in order to quickly adapt to novel environments that share a common underlying structure. Here we combine functional imaging and computational modeling to identify the neural systems that support the discovery and transfer of hierarchical task structure. Human subjects completed multiple blocks of a reinforcement learning task that contained a global hierarchical structure governing stimulus-response action mapping. First, behavioral and computational evidence showed that humans successfully discover and transfer the hierarchical rule structure embedded within the task. Next, analysis of fMRI BOLD data revealed activity across a frontal-parietal network that was specifically associated with the discovery of this embedded structure. Finally, activity throughout a cingulo-opercular network and in caudal frontal cortex supported the transfer and implementation of this discovered structure. Together, these results reveal a division of labor in which dissociable neural systems support the learning and transfer of abstract control structures..

Neural representation of abstract task structure during generalization

“…Frontoparietal systems may be more involved in representing this kind of abstract task structure when it is necessary for actively controlling behavior contingent on changing contexts, as in this experiment. Other work that has examined analogous abstract task representations in the context of conditional action selection and selecting between causal models have found evidence for the involvement of frontoparietal cortex ( Collins et al, 2014 ; Eichenbaum et al, 2020 ; Tomov et al, 2018 ), similar to our results. In contrast, OFC and HPC could be engaged more by resolving uncertainty about the prevailing latent task state ( Chan et al, 2016 ; Saez et al, 2015 ), or planning within a cognitive map of latent task states ( Schuck et al, 2016 ; Zhou et al, 2019 ).…”

“…Like hierarchical task structure in those experiments, latent task states enable faster learning, reduced memory load, and greater behavioral flexibility ( Frank and Badre, 2012 ; Koechlin and Summerfield, 2007 ). Recent work has similarly implicated mid-lateral and inferior parietal cortex in inferring latent causes within tasks ( Tomov et al, 2018 ) and discovering hierarchical rules ( Collins et al, 2014 ; Eichenbaum et al, 2020 ). Our findings extend these observations to indicate that this network is not just involved in discovering and maintaining task structure, but more generally represents tasks in an abstract format that facilitates novel behaviors, even in the absence of reinforcement.…”

“…Recent work has provided evidence that the orbitofrontal cortex (OFC) and hippocampus (HPC) maintain structured, abstract representations of tasks during planning and navigation in conceptual spaces, including latent task states ( Liu et al, 2019 ; McKenzie et al, 2014 ; Schuck et al, 2016 ; Tavares et al, 2015 ; Wilson et al, 2014 ; Zhou et al, 2019 ). Likewise, other studies have investigated the neural processes supporting fast acquisition of stimulus-response rules based on latent task knowledge ( Badre et al, 2010 ; Collins et al, 2014 ; Eichenbaum et al, 2020 ; Frank and Badre, 2012 ). However, an essential feature of an abstract representation is that it can be used to generalize behaviors to new settings, in absence of feedback, through a process of inference.…”

Neural representation of abstract task structure during generalization

“…Like hierarchical task structure in those experiments, latent task states enable faster learning, reduced memory load and greater behavioral flexibility (Frank and Badre, 2012; Koechlin and Summerfield, 2007). Recent work has similarly implicated mid-lateral and inferior parietal cortex in inferring latent causes within tasks (Tomov et al, 2018), and discovering hierarchical rules (Collins et al, 2014; Eichenbaum et al, 2020). Our findings extend these observations to indicate that this network is not just involved in discovering and maintaining task structure, but more generally represents tasks in an abstract format that facilitates novel behaviors, even in absence of reinforcement.…”

Frontopolar Cortex Specializes for Manipulation of Structured Information

“…Recently, several researchers have proposed a caudal-rostral organization of function in the frontal lobes, with the most complex or abstract information processing found in the most anterior portion of frontal cortex ( Krawczyk et al, 2011 ; Voytek et al, 2015 ; Nee and D Esposito, 2016 ; Dixon et al, 2017 ; Duverne and Koechlin, 2017 ; Badre and Nee, 2018 ; Jerath et al, 2019 ; Sarafyazd and Jazayeri, 2019 ; Eichenbaum et al, 2020 ; Riddle et al, 2020 ). The nature of the cognitive function employed in these studies has not been uniform.…”