Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning

Viejo, Guillaume; Khamassi, Mehdi; Brovelli, Andrea; Girard, Benoît

doi:10.3389/fnbeh.2015.00225

Cited by 51 publications

(102 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This allowed the validation of the model, in 165 particular to check if the learning processes of the world model lead to match human 166 performance, and if the arbitration mechanism employing a variable time planning lead 167 to reproduce the reaction times exhibited by human participants. This target 168 experiment was also investigated by the model proposed in [38]. Although this model 169 did not aim to bridge probabilistic modelling to neural mechanisms as here, it used an 170 interesting mechanism of arbitration between goal-directed and habitual behaviour 171 based on entropy as done here.…”

mentioning

confidence: 99%

“…Curves indicate the values averaged over 14 human participants and 20 artificial participants; error bars indicate the standard error. The data of human participants are from [38].…”

mentioning

confidence: 99%

“…Indeed, 565 Fig 5. Comparison of the reaction times of the humans and simulated participants. (A) Reaction times of human participants averaged over S1, S2, and S3 (y-axis) for the 'representative steps' (x-axis); the 'representative steps' allow the alignment of the reaction times of the three stimuli so as to separate the exploration phase (first 5 steps) and the exploitation phase (6 steps onward); to this purpose, the reaction times for S1 obtained in succeeding trials from the first onward is assigned the steps (used to compute the averages shown in the plot) '1, 2, 6, 7, ...', whereas S2 is assigned the steps '1, 2, 3, 4, 6, 7, ...', and S3 is assigned the steps '1, 2, 3, 4, 5, 6, 7, ...'; data are taken from [38]; (B) Reaction times of the model plotted in the same way.…”

mentioning

confidence: 99%

“…Another model [38] used an entropy-based measure as a means to decide to give hypotheses deserve further study.…”

mentioning

confidence: 99%

“…Another difference of our model with respect to the models of [37,38] is that it uses 634 arbitration to select between exploration (attempts of different actions serving the 635 world-model learning) vs. exploitation of the acquired knowledge (planning), rather than 636 between goal-directed behaviour and habitual behaviour. This was done as habitual 637 behaviour takes long to form and so it seems to be ruled out during the first attempts 638 to solve new problems [3].…”

mentioning

confidence: 99%

See 4 more Smart Citations

A spiking neural-network model of goal-directed behaviour

Basanisi

Brovelli

Cartoni

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Idea of the model, specification of the model and tests, implementation of the model, tests, data analysis, analysis of results, writing-up. ‡Idea of the model, specification of the model and tests, analysis of results, writing-up. ¤Specification of the model and tests, analysis of results, writing-up. * AbstractIn mammals, goal-directed and planning processes support flexible behaviour usable to face new situations or changed conditions that cannot be tackled through more efficient but rigid habitual behaviours. Within the Bayesian modelling approach of brain and behaviour, probabilistic models have been proposed to perform planning as a probabilistic inference. Recently, some models have started to face the important challenge met by this approach: grounding such processes on the computations implemented by brain spiking networks. Here we propose a model of goal-directed behaviour that has a probabilistic interpretation and is centred on a recurrent spiking neural network representing the world model. The model, building on previous proposals on spiking neurons and plasticity rules having a probabilistic interpretation, presents these novelties at the system level: (a) the world model is learnt in parallel with its use for planning, and an arbitration mechanism decides when to exploit the world-model knowledge with planning, or to explore, on the basis of an entropy-based confidence on the world model knowledge; (b) the world model is a hidden Markov model (HMM) able to simulate sequences of states and actions, thus planning selects actions through the same neural generative process used to predict states; (c) the world model learns the hidden causes of observations, and their temporal dependencies, through a biologically plausible unsupervised learning mechanism. The model is tested with a visuomotor learning task and validated by comparing its behaviour with the performance and reaction times of human participants solving the same task. The model represents a further step towards the construction of an autonomous architecture bridging goal-directed behaviour as probabilistic inference to brain-like computations. Author summaryGoal-directed behaviour relies on brain processes supporting planning of actions based on the prediction of their consequences before performing them in the environment. An important computational modelling approach of these processes sees the brain as a probabilistic machine implementing goal-directed processes relying on probability distributions and operations on them. An important challenge for this approach is to explain how these distributions and operations might be grounded on the brain spiking December 2, 2019 1/28 neurons and learning processes. Here we propose a hypothesis of how this might happen by presenting a computational model of goal-directed processes based on artificial spiking neural networks. The model presents three main novelties. First, it can plan even while it is still learning the consequences of actions by deciding if planning or exploring the environment bas...

show abstract

mentioning

confidence: 99%

“…Curves indicate the values averaged over 14 human participants and 20 artificial participants; error bars indicate the standard error. The data of human participants are from [38].…”

mentioning

confidence: 99%

mentioning

confidence: 99%

“…Another model [38] used an entropy-based measure as a means to decide to give hypotheses deserve further study.…”

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

A spiking neural-network model of goal-directed behaviour

Basanisi

Brovelli

Cartoni

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

Colas

Dundon

Gerraty

et al. 2022

Human Brain Mapping

View full text Add to dashboard Cite

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single rewardprediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversallearning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With highresolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

show abstract

Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Khamassi

2020

Theory and Practice of Natural Computing

Self Cite

View full text Add to dashboard Cite

Engineering approaches to machine learning (including robot learning) typically seek for the best learning algorithm for a particular problem, or a set problems. In contrast, the mammalian brain appears as a toolbox of different learning strategies, so that any newly encountered situation can be autonomously learned by an animal with a combination of existing learning strategies. For example, when facing a new navigation problem, a rat can either learn a map of the environment and then plan to find a path to its goal within this map. Alternatively, it can learn sequences of egocentric movements in response to identifiable features of the environment. For about 15 years, computational neuroscientists have searched for the mammalian brain's coordination mechanisms which enable it to find efficient, if not necessarily optimal, combinations of existing learning strategies to solve new problems. Understanding such coordination principles of multiple learning strategies could have great implications in robotics, to enable robots to autonomously determine which learning strategies are appropriate in different contexts. Here, we review some of the main neuroscience models for the coordination of learning strategies and present some of the early results obtained when applying these models to robot learning. We moreover highlight important energy costs which can be reduced with such bio-inspired solutions compared to current deep reinforcement learning approaches. We conclude by sketching a roadmap for further developing such bio-inspired hybrid learning approaches to robotics.

show abstract

Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning

Cited by 51 publications

References 65 publications

A spiking neural-network model of goal-directed behaviour

A spiking neural-network model of goal-directed behaviour

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Contact Info

Product

Resources

About