Reinforcement learning with Marr

Niv, Yaron; Langdon, Angela J.

doi:10.1016/j.cobeha.2016.04.005

Cited by 45 publications

(29 citation statements)

References 89 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This can explain the experimentally observed ramping DA signals [ 5 – 8 ] as reflecting a gradient of state values created by temporal discounting (as in our Fig 6A and 6B ), also consistent with the arguments by [ 7 ]. These normative hypotheses, at the Marr's levels of computation and algorithm [ 44 , 45 ], provide intriguing predictions that are desired to be experimentally tested. Meanwhile, it is also important to explore the Marr's level of implementation, namely, circuit/synaptic operations, which could potentially provide inspirations for the upper levels and vice versa [ 45 ].…”

Section: Discussionmentioning

confidence: 99%

“…These normative hypotheses, at the Marr's levels of computation and algorithm [ 44 , 45 ], provide intriguing predictions that are desired to be experimentally tested. Meanwhile, it is also important to explore the Marr's level of implementation, namely, circuit/synaptic operations, which could potentially provide inspirations for the upper levels and vice versa [ 45 ]. The abovementioned normative hypotheses highlight essential issues at the circuit/synaptic level, including how the sustained DA signals are generated in the upstream and utilized in the downstream, how the selection of action timing is implemented, and how temporal discounting is implemented.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

Kato

Morita

2016

PLoS Comput Biol

View full text Add to dashboard Cite

It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of ‘Go’ or ‘No-Go’ selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of ‘Go’ values towards a goal, and (2) value-contrasts between ‘Go’ and ‘No-Go’ are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

Kato

Morita

2016

PLoS Comput Biol

View full text Add to dashboard Cite

show abstract

“…We note that this is the opposite of the way value functions are usually thought of in neuroscience, which stresses the primacy of the value function from which policies are derived. However, there is experimental evidence that signals in the striatum are more suitable for direct policy search rather than for updating action values as an intermediate step, as would be the case for value function-based approaches to computing the policy (Li & Daw, 2011;Niv & Langdon, 2016). Moreover, although we have used a single RNN each to represent the policy and value modules, using "deep," multilayer RNNs may increase the representational power of each module (Pascanu et al, 2013a).…”

Section: Discussionmentioning

confidence: 99%

Reward-based training of recurrent neural networks for cognitive and value-based tasks

Song

Yang

Wang

2016

Preprint

View full text Add to dashboard Cite

Trained neural network models, which exhibit many features observed in neural recordings from behaving animals and whose activity and connectivity can be fully analyzed, may provide insights into neural mechanisms. In contrast to commonly used methods for supervised learning from graded error signals, however, animals learn from reward feedback on definite actions through reinforcement learning. Reward maximization is particularly relevant when the optimal behavior depends on an animal's internal judgment of confidence or subjective preferences. Here, we describe reward-based training of recurrent neural networks in which a value network guides learning by using the selected actions and activity of the policy network to predict future reward. We show that such models capture both behavioral and electrophysiological findings from well-known experimental paradigms. Our results provide a unified framework for investigating diverse cognitive and value-based computations, including a role for value representation that is essential for learning, but not executing, a task.

show abstract

“…The actor is in charge of choosing an action in a given state (policy) while the critic is in charge of evaluating (criticizing) the current state (value function). This classical view has been used extensively for modelling the basal ganglia (Suri, R E & Schultz, W, 1999; Suri, 2002; Frank, 2004; Doya, 2007; Glimcher, 2011; Doll, Bradley B, Simon, Dylan A, & Daw, Nathaniel D, 2012) even though the precise anatomical mapping of these two components is still subject to debate and may diverge from one model to the other (Redgrave, Peter, Gurney, Kevin, & Reynolds, John, 2008; Niv, Yael & Langdon, Angela, 2016). However, all these models share the implicit assumption that the actor and the critic are acting in concert, i.e.…”

Section: Introductionmentioning

confidence: 99%

Dual Competition between the Basal Ganglia and the Cortex: from Action-Outcome to Stimulus-Response

Topalidou

Kase

Boraud

et al. 2017

Preprint

View full text Add to dashboard Cite

Action-outcome (A-O) and stimulus-response (S-R) processes that are two forms of instrumental conditioning that are important components of decision making and action selection. The former adapts its response according to the outcome while the latter is insensitive to the outcome. An unsolved question is how these two processes emerge, cooperate and interact inside the brain in order to issue a unique behavioral answer. Here we propose a model of the interaction between the cortex, the basal ganglia and the thalamus based on a dual competition. We hypothesize that the striatum, the subthalamic nucleus, the internal pallidum (GPi), the thalamus, and the cortex are involved in closed feedback loops through the hyperdirect and direct pathways. These loops support a competition process that results in the ability for the basal ganglia to make a cognitive decision followed by a motor decision. Considering lateral cortical interactions (short range excitation, long range inhibition), another competition takes place inside the cortex allowing this latter to make a cognitive and a motor decision. We show how this dual competition endows the model with two regimes. One is oriented towards action-outcome and is driven by reinforcement learning, the other is oriented towards stimulus-response and is driven by Hebbian learning. The final decision is made according to a combination of these two mechanisms with a gradual transfer from the former to the latter. We confirmed these theoretical results on primates using a two-armed bandit task and a reversible bilateral inactivation of the internal part of the globus pallidus.

show abstract

Reinforcement learning with Marr

Cited by 45 publications

References 89 publications

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

Reward-based training of recurrent neural networks for cognitive and value-based tasks

Dual Competition between the Basal Ganglia and the Cortex: from Action-Outcome to Stimulus-Response

Contact Info

Product

Resources

About