Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework

Abstract

We review the psychology and neuroscience of reinforcement learning (RL), which has witnessed significant progress in the last two decades, enabled by the comprehensive experimental study of simple learning and decision-making tasks. However, the simplicity of these tasks misses important aspects of reinforcement learning in the real world: (i) State spaces are high-dimensional, continuous, and partially observable; this implies that (ii) data are relatively sparse: indeed precisely the same situation may never be encountered twice; and also that (iii) rewards depend on long-term consequences of actions in ways that violate the classical assumptions that make RL tractable. A seemingly distinct challenge is that, cognitively, these theories have largely connected with procedural and semantic memory: how knowledge about action values or world models extracted gradually from many experiences can drive choice. This misses many aspects of memory related to traces of individual events, such as episodic memory. We suggest that these two gaps are related. In particular, the computational challenges can be dealt with, in part, by endowing RL systems with episodic memory, allowing them to (i) efficiently approximate value functions over complex state spaces, (ii) learn with very little data, and (iii) bridge long-term dependencies between actions and rewards. We review the computational theory underlying this proposal and the empirical evidence to support it. Our proposal suggests that the ubiquitous and diverse roles of memory in RL may function as part of an integrated learning system.

Predicting the past, remembering the future

“…However, more recent experiments have found semantic impairments in amnesia (see [72] for a review), and amnesics exhibit semantically impoverished episodic future-thinking [73]. Beyond semantic memory, there is now considerable evidence that the hippocampus (and episodic memory more generally) plays an important role in reinforcement learning as well [7]. Collectively, these observations lend credence to the idea that multiple forms of memory may draw upon a common predictive representation in the hippocampus.…”

“…At shorter timescales, many events can be anticipated using sensory information from the recent past, giving organisms with memory an adaptive advantage. Thus, memory of the past, spanning vastly different timescales, is relevant for predicting and controlling the future [4, 5, 6, 7]. …”

Planning and navigation as active inference

“…To meet this challenge, a recent proposal endowed reinforcement learning systems with episodic memory (Gershman and Daw 2017). Interestingly, this approach produces equations that are formally similar to the updates in active inference (because they both involve message passing among representations of hidden or latent states in the future and past).…”

Ventral striatum lesions do not affect reinforcement learning with deterministic outcomes on slow time scales.

“…In addition to reinforcement learning and working memory, episodic memory could also contribute to learning in our tasks (Gershman & Daw, 2017; Lengyel & Dayan, 2007). It has been proposed that, when state spaces are large or continuous, and little data is available, neither model based nor model free learning mechanisms can effectively mediate behavior.…”

“…Alternatively, after receiving feedback on the outcome of a given trial, monkeys may have used prospective coding of the choice to be made on the next trial in combination with working memory, a phenomenon thought to underlie learning set formation (Murray & Gaffan, 2006). In addition to working memory and prospective coding explanations, the animals may have also been using some form of episodic memory to store outcomes (Gershman & Daw, 2017). …”