Learning task-state representations

Abstract

Arguably, the most difficult part of learning is deciding what to learn about. Should I associate the positive outcome of safely completing a street-crossing with the situation "the car approaching the crosswalk was red" or "the approaching car was slowing down"? In this Perspective, we summarize our recent research into the computational and neural underpinnings of "representation learning"-how humans (and other animals) construct task representations that allow efficient learning and decision making. We first discuss the problem of learning what to ignore when confronted with too much information, so that experience can properly generalize across situations. We then turn to the problem of augmenting perceptual information with inferred latent causes that embody unobservable task-relevant information such as contextual knowledge. Finally, we discuss recent findings regarding the neural substrates of task representations that suggest the orbitofrontal cortex represents "task states," deploying them for decision-making and learning elsewhere in the brain. The ubiquitous problem of representation learning Imagine standing on a street corner and preparing to cross the street on your way home (Figure 1A). Even in the calmest of neighborhoods, your sensory systems will confront a staggering amount of information that may or may not be relevant for the decision of whether to go or to wait. Computationally, avoiding getting run over is daunting. Nevertheless, you can probably complete the street-crossing task successfully even while talking to a friend or mentally planning your afternoon. What allows our brains to make decisions in complex, multidimensional environments with such ease and efficiency? We argue that the brain solves seemingly complex tasks by learning efficient, low dimensional representations that simplify these tasks. A useful task representation will focus on aspects of the environment that are critical to correct performance of the task, that is, it will include all factors that are causally related to the outcome of our actions, for instance, the speed and distance of the closest oncoming car. At the same time, the task representation will gloss over all other information: the colors of the cars, the shops across the street, etc. Ignoring input dimensions (color, shape) that are irrelevant for task performance and concentrating on the few dimensions that are critical (speed, distance) allows us not only to make rapid decisions, but also to generalize learning as widely as possible. That is, correctly ignoring irrelevant aspects of the environment will allow learning from one experience to inform decision making in other scenarios that share relevant features with the current experience and differ only in the irrelevant ones.

Private-public mappings in human prefrontal cortex

“…Context may also hold structural information that can be generalised across events (e.g., many behaviours which are acceptable at home are not acceptable when visiting others) (Gershman, 2017). There is evidence that such structural knowledge is encoded in the medial temporal lobe and ventral PFC (Behrens et al, 2018;Niv, 2019). Our results complement this work by identifying representations in FPl that may support a mapping between private states and public actions that are appropriate for the current context.…”

“…Recent years has seen an interest in how the brain encodes structured representations of the world for efficient learning and decision-making (Behrens et al, 2018;Niv, 2019;Radulescu, Niv, & Ballard, 2019). In these scenarios, context may be particularly important for adaptive behaviour.…”

Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems

“…The precise computations that mediate model-contingent credit assignment depend on the brain's internal representation of the task state-space [2][3][4]32,33 . In many laboratory tasks, including ours, more than one task representation is possible 2,33 .…”

“…The precise computations that mediate model-contingent credit assignment depend on the brain's internal representation of the task state-space [2][3][4]32,33 . In many laboratory tasks, including ours, more than one task representation is possible 2,33 . However, to benefit from the relational knowledge, any such representation must differentiate the related stimuli (A,B) from the control stimulus (C) so that AB outcomes can cause learning on both A and B but not C. For example, one possible representation is to maintain a single variable encoding the state of the AB pair 2 and a second variable encoding the state of C.…”

Preparation for upcoming attentional states in the hippocampus and medial prefrontal cortex

“…In rodents, increased neural synchrony between the hippocampus and mPFC has been observed at decision points in which memory must be used to guide future behavior (Benchenane et al, 2010;Jones & Wilson, 2005). In humans, functional magnetic resonance imaging (fMRI) studies have demonstrated that the orbitofrontal cortex (a region in the ventral medial prefrontal cortex) represents goal state representations that are not explicitly instructed but rather inferred on the basis of past experience (Niv, 2019;Schuck, Cai, Wilson, & Niv, 2016;Schuck et al, 2015). Moreover, the hippocampus and ventromedial PFC (vmPFC) show functional coupling as individuals learn which features of an object are relevant for determining its category, and thus should be attended (Mack et al, 2016).…”