Recurrent Spiking Networks Solve Planning Tasks

Rueckert, Elmar; Kappel, David; Tanneberg, Daniel; Pecevski, Dejan; Peters, Jan

doi:10.1038/srep21142

Cited by 56 publications

(92 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This idea is in line with recent proposals in machine learning, [9] and [10], that use also semi-structured recurrent network models for planning. In comparison to them, we extend their results by adding a second structure along with the recurrent network, an associative map (AM), that will recursively and timely control it; see Fig 1b).…”

Section: Introductionsupporting

confidence: 65%

Iterative free-energy optimization for recurrent neural networks (INFERNO)

Pitti¹,

Gaussier²,

Quoy³

2017

PLoS ONE

View full text Add to dashboard Cite

The intra-parietal lobe coupled with the Basal Ganglia forms a working memory that demonstrates strong planning capabilities for generating robust yet flexible neuronal sequences. Neurocomputational models however, often fails to control long range neural synchrony in recurrent spiking networks due to spontaneous activity. As a novel framework based on the free-energy principle, we propose to see the problem of spikes’ synchrony as an optimization problem of the neurons sub-threshold activity for the generation of long neuronal chains. Using a stochastic gradient descent, a reinforcement signal (presumably dopaminergic) evaluates the quality of one input vector to move the recurrent neural network to a desired activity; depending on the error made, this input vector is strengthened to hill-climb the gradient or elicited to search for another solution. This vector can be learned then by one associative memory as a model of the basal-ganglia to control the recurrent neural network. Experiments on habit learning and on sequence retrieving demonstrate the capabilities of the dual system to generate very long and precise spatio-temporal sequences, above two hundred iterations. Its features are applied then to the sequential planning of arm movements. In line with neurobiological theories, we discuss its relevance for modeling the cortico-basal working memory to initiate flexible goal-directed neuronal chains of causation and its relation to novel architectures such as Deep Networks, Neural Turing Machines and the Free-Energy Principle.

show abstract

Section: Introductionsupporting

confidence: 65%

Iterative free-energy optimization for recurrent neural networks (INFERNO)

Pitti¹,

Gaussier²,

Quoy³

2017

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…These and other formal schemes may be used to test empirical predictions on how rodents solve challenging navigation tasks or even how humans solve abstract tasks . Compared with other computational proposals that also discuss hippocampal function in relation to generative models, we have stressed the idea that hippocampal coding and processing may be organized around sequences. Hippocampal generative models may be preconfigured for sequential processing and the rapid encoding of arbitrary sequences of events in ways that current machine learning techniques fail to model.…”

Section: Discussionmentioning

confidence: 99%

“…Finally, a series of computational studies explored the idea that various aspects of spatial cognition, including spatial decision making, route planning, model selection, vicarious trial and error (VTE), and the covert evaluation of future spatial trajectories, may be based on probabilistic inference and a common generative model (implemented in the hippocampus and surrounding structures), also discussing various neuronal implementations of (approximate) Bayesian inference …”

Section: A Computational Perspective On Igssmentioning

confidence: 99%

“…86 Finally, a series of computational studies explored the idea that various aspects of spatial cognition, including spatial decision making, route planning, model selection, vicarious trial and error (VTE), and the covert evaluation of future spatial trajectories, may be based on probabilistic inference and a common generative model (implemented in the hippocampus and surrounding structures), also discussing various neuronal implementations of (approximate) Bayesian inference. 7,75,79,87,88 By leveraging and extending these and other models, we discuss below what the notions of statistical inference and generative models can offer to the study of prospective cognition and the relations between stimulus-evoked and internally generated brain activity as applied to theta and SWR sequences.…”

Section: A Computational Perspective On Igssmentioning

confidence: 99%

See 1 more Smart Citation

Internally generated hippocampal sequences as a vantage point to probe future‐oriented cognition

Pezzulo

Kemere

Meer

2017

Annals of the New York Academy of Sciences

View full text Add to dashboard Cite

Information processing in the rodent hippocampus is fundamentally shaped by internally generated sequences (IGSs), expressed during two different network states: theta sequences, which repeat and reset at the ∼8 Hz theta rhythm associated with active behavior, and punctate sharp wave-ripple (SWR) sequences associated with wakeful rest or slow-wave sleep. A potpourri of diverse functional roles has been proposed for these IGSs, resulting in a fragmented conceptual landscape. Here, we advance a unitary view of IGSs, proposing that they reflect an inferential process that samples a policy from the animal's generative model, supported by hippocampus-specific priors. The same inference affords different cognitive functions when the animal is in distinct dynamical modes, associated with specific functional networks. Theta sequences arise when inference is coupled to the animal's action-perception cycle, supporting online spatial decisions, predictive processing, and episode encoding. SWR sequences arise when the animal is decoupled from the action-perception cycle and may support offline cognitive processing, such as memory consolidation, the prospective simulation of spatial trajectories, and imagination. We discuss the empirical bases of this proposal in relation to rodent studies and highlight how the proposed computational principles can shed light on the mechanisms of future-oriented cognition in humans.

show abstract

“…At the low level, the 125 model relies on the neural mechanisms and the unsupervised learning plasticity rule 126 December 2, 2019 4/28 proposed in [24] (plus a reinforcement learning rule) to implement a HMM with a 127spiking recurrent neural network. A recurrent spiking neural network to implement the 128 world model and planning was also used in [35,36]. With respect to these previous 129 models, our architecture presents a number of structure and functioning novelties at the 130 system level.…”

mentioning

confidence: 99%

A spiking neural-network model of goal-directed behaviour

Basanisi

Brovelli

Cartoni

et al. 2019

Preprint

View full text Add to dashboard Cite

Idea of the model, specification of the model and tests, implementation of the model, tests, data analysis, analysis of results, writing-up. ‡Idea of the model, specification of the model and tests, analysis of results, writing-up. ¤Specification of the model and tests, analysis of results, writing-up. * AbstractIn mammals, goal-directed and planning processes support flexible behaviour usable to face new situations or changed conditions that cannot be tackled through more efficient but rigid habitual behaviours. Within the Bayesian modelling approach of brain and behaviour, probabilistic models have been proposed to perform planning as a probabilistic inference. Recently, some models have started to face the important challenge met by this approach: grounding such processes on the computations implemented by brain spiking networks. Here we propose a model of goal-directed behaviour that has a probabilistic interpretation and is centred on a recurrent spiking neural network representing the world model. The model, building on previous proposals on spiking neurons and plasticity rules having a probabilistic interpretation, presents these novelties at the system level: (a) the world model is learnt in parallel with its use for planning, and an arbitration mechanism decides when to exploit the world-model knowledge with planning, or to explore, on the basis of an entropy-based confidence on the world model knowledge; (b) the world model is a hidden Markov model (HMM) able to simulate sequences of states and actions, thus planning selects actions through the same neural generative process used to predict states; (c) the world model learns the hidden causes of observations, and their temporal dependencies, through a biologically plausible unsupervised learning mechanism. The model is tested with a visuomotor learning task and validated by comparing its behaviour with the performance and reaction times of human participants solving the same task. The model represents a further step towards the construction of an autonomous architecture bridging goal-directed behaviour as probabilistic inference to brain-like computations. Author summaryGoal-directed behaviour relies on brain processes supporting planning of actions based on the prediction of their consequences before performing them in the environment. An important computational modelling approach of these processes sees the brain as a probabilistic machine implementing goal-directed processes relying on probability distributions and operations on them. An important challenge for this approach is to explain how these distributions and operations might be grounded on the brain spiking December 2, 2019 1/28 neurons and learning processes. Here we propose a hypothesis of how this might happen by presenting a computational model of goal-directed processes based on artificial spiking neural networks. The model presents three main novelties. First, it can plan even while it is still learning the consequences of actions by deciding if planning or exploring the environment bas...

show abstract

Recurrent Spiking Networks Solve Planning Tasks

Cited by 56 publications

References 57 publications

Iterative free-energy optimization for recurrent neural networks (INFERNO)

Iterative free-energy optimization for recurrent neural networks (INFERNO)

Internally generated hippocampal sequences as a vantage point to probe future‐oriented cognition

A spiking neural-network model of goal-directed behaviour

Contact Info

Product

Resources

About