Vector-based navigation using grid-like representations in artificial agents

Abstract

Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go. Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex . Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space and is critical for integrating self-motion (path integration) and planning direct trajectories to goals (vector-based navigation). Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types . We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments-optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation, demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments.

Neurobiological successor features for spatial navigation

“…This form of eigendecomposition is similar to other dimensionality reduction techniques that have been used to generate grid cells from populations of idealised place cells (Dordek, Soudry, Meir, & Derdikman, 2016). Previously, low dimensional encodings such as these have been show to accelerate learning and facilitate vector-based navigation (Gustafson & Daw, 2011;Banino et al, 2018).…”

Dynamic compression and expansion in a classifying recurrent network

“…We employ the most basic and widely used training rule for neural networks: stochastic gradient descent with backpropagation (through time). While not biologically plausible in its simplest form, the characteristics of networks trained by this algorithm can still resemble dynamics in the brain [38,53,4,41,5].…”

An uncertainty principle for neural coding: Conjugate representations of position and velocity are mapped onto firing rates and co‐firing rates of neural spike trains

“…If this could be achieved, then it would be intriguing to investigate what kinds of coding mechanisms naturally emerge when a recurrent network of sigma‐chi neurons is trained to perform path integration. Phenomena such as oscillatory rhythms that support dial coding might emerge spontaneously from the training of such a network, in a manner similar to the way that grid cells emerge spontaneously when a network of linear neurons is trained to perform path integration (Banino et al, ). The emergent firing properties and connectivity patterns in a trained network of recurrently connected sigma‐chi neurons might bear closer resemblance to real hippocampal and entorhinal networks than those that emerge from networks of linear neurons.…”

“…Reservoir computing models of path integration are recurrent neural networks trained from example data (via gradient descent methods) to convert time‐varying velocity signals into time‐varying position signals (Abbott, DePasquale, & Memmesheimer, ; Denève & Machens, ). It has recently been shown that neurons with periodic spatial tuning—similar to entorhinal grid cells—can emerge spontaneously in a nonspiking recurrent network trained to perform spatial path integration (Banino et al, ). Like attractor models, most reservoir computing models of path integration use recurrent networks composed from linear neurons, so the inputs and outputs to these networks encode velocity and position signals solely as vectors of neural firing rates (not spike train correlations).…”