Learning Structures: Predictive Representations, Replay, and Generalization

Abstract

Memory and planning rely on learning the structure of relationships among experiences. Compact representations of these structures guide flexible behavior in humans and animals. A century after 'latent learning' experiments summarized by Tolman, the larger puzzle of cognitive maps remains elusive: how does the brain learn, generalize, and transfer relational structures? This review focuses on a reinforcement learning (RL) approach to learning compact representations of the structure of states. We review evidence showing that capturing structures as predictive representations updated via replay offers a neurally plausible account of human behavior and the neural representations of predictive cognitive maps. We highlight multi-scale successor representations, prioritized replay, and policy-dependence. These advances call for new directions in studying the entanglement of learning and memory with prediction and planning.

Predictive Maps in Rats and Humans for Spatial Navigation

“…Our finding build upon recent work showing that the actions of human participants are best explained by a combination of SR and model-based behaviours . Since the SR encodes the environment's transition structure, it is itself a model that can be leveraged for intuitive planning (Baram et al, 2018) or more explicit planning procedures such as a tree search -which may also provide a function for hippocampal replay (Mattar & Daw, 2018;Ida Momennejad, 2020;Ida Momennejad et al, 2018;Ólafsdóttir et al, 2017;Pfeiffer & Foster, 2013). Our work also adds to findings that the SR provides an account of hippocampal representations observed in rats and humans (Brunec & Momennejad, 2019;de Cothi & Barry, 2020;Garvert et al, 2017;Stachenfeld et al, 2017) by showing it also fits their spatial navigation behaviour in dynamic environments.…”

Striatal and hippocampal contributions to flexible navigation in rats and humans

“…Interestingly, components of the successor representation during simulations show similarities to properties of place cells and grid cells, including the influence of goal locations on place field over-representation observed in specific paradigms and influence of environmental geometry on grid field integrity ( Duvelle et al, 2019 ; Ekstrom et al, 2020 ; Krupic et al, 2015 ; Stachenfeld et al, 2017 ). It is an interesting future direction for studies to investigate the relationship between neural responses and the internal computations of successor representation shown to account for behaviour flexibility particularly in some spatial navigation tasks ( Russek et al, 2017 ; for review see Momennejad, 2020 ). Recent work with rats navigating between four interconnected rooms has revealed that during initial adaptation to pathways being obstructed place cells in CA1 do not adapt their firing fields to accompany the changing behaviour ( Duvelle et al, 2020 ) as might have been predicted by a model in which place cells support SR coding ( Stachenfeld et al, 2017 ).…”

“…detours that require larger or smaller shifts in the route to the goal). It would also be important to examine the interplay between the striatum, hippocampal/parahippocampal structures, and prefrontal cortex during such updating and representation for the structure of the environment (see Momennejad, 2020 ). The entorhinal cortex has also been proposed to play a role in coding the transition structure of the layout of the environment or stimulus set ( Behrens et al, 2018 ).…”

Predictive Representations in Hippocampal and Prefrontal Hierarchies

“…In a deterministic environment, when there is a transition from a given state S i to state S j , we assign 1 in the ith row and jth column of T ( Supplementary Figure 1, left). The successor representation under a random policy can be then computed from T as follows (for comparison to policy-dependent SR see Momennejad, 2020): 2expands equation 1 for computing the successor representation from state s 1 to the goal state s g from T, which is one cell in the SR matrix. Recall that T denotes the matrix of one-step transition probabilities among adjacent states, while SR contains multi-step dependencies among non-adjacent states.…”