Tamping Ramping: Algorithmic, Implementational, and Computational Explanations of Phasic Dopamine Signals in the Accumbens

Lloyd, Kevin; Dayan, Peter

doi:10.1371/journal.pcbi.1004622

Cited by 47 publications

(45 citation statements)

References 130 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This time course suggests that prediction error signals reflect the evolving decision process. Given the dense dopaminergic projections to brain regions involved in decision making [48], the early dopamine prediction errors might even be able to influence the current choice computation, for instance by modulating the gain of evidence accumulation [49]. Alternatively, pre-decision dopamine responses do not impact choices directly but other aspects of immediate behavior such as the willingness to complete the trial [50].…”

Section: Discussionmentioning

confidence: 99%

Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision

et al. 2017

View full text Add to dashboard Cite

Summary Central to the organization of behavior is the ability to predict the values of outcomes to guide choices. The accuracy of such predictions is honed by a teaching signal that indicates how incorrect a prediction was (‘reward prediction error’, RPE). In several reinforcement learning contexts such as Pavlovian conditioning and decisions guided by reward history, this RPE signal is provided by midbrain dopamine neurons. In many situations, however, the stimuli predictive of outcomes are perceptually ambiguous. Perceptual uncertainty is known to influences choices, but it has been unclear whether or how dopamine neurons factor it into their teaching signal. To cope with uncertainty, we extended a reinforcement learning model with a belief state about the perceptually ambiguous stimulus; this model generates an estimate of the probability of choice correctness, termed decision confidence. We show that dopamine responses in monkeys performing a perceptually ambiguous decision task comply with the model’s predictions. Consequently, dopamine responses did not simply reflect a stimulus’ average expected reward value, but were predictive of the trial-to-trial fluctuations in perceptual accuracy. These confidence-dependent dopamine responses emerged prior to monkeys’ choice initiation raising the possibility that dopamine impacts impeding decisions, in addition to encoding a post-decision teaching signal. Finally, by manipulating reward size, we found that dopamine neurons reflect both the upcoming reward size and the confidence in achieving it. Together, our results show that dopamine responses convey teaching signals that are also appropriate for perceptual decisions.

show abstract

Section: Discussionmentioning

confidence: 99%

Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Under many conditions, however, it has been difficult to disambiguate whether dopamine represents RPE or value. First, a dopamine ramp can occur regardless of whether dopamine represents value or RPEs (Gershman, 2014;Lloyd and Dayan, 2015;Morita and Kato, 2014). For instance, a theoretical study showed that the shape of the value function matters: if the value function is a convex function of proximity to reward, a TD RPE can exhibit a positive ramp (Gershman, 2014) (more general conditions are described in the Supplementary Note; see also Fig.…”

Section: Introductionmentioning

confidence: 99%

A unified framework for dopamine signals across timescales

Kim

Malik

Mikhael

et al. 2019

Preprint

View full text Add to dashboard Cite

Rapid phasic activity of midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), resembling temporal difference errors used in machine learning. Recent studies describing slowly increasing dopamine signals have instead proposed that they represent state values and arise independently from somatic spiking activity. Here, we developed novel experimental paradigms using virtual reality that disambiguate RPEs from values. We examined the dopamine circuit activity at various stages including somatic spiking, axonal calcium signals, and striatal dopamine concentrations. Our results demonstrate that ramping dopamine signals are consistent with RPEs rather than value, and this ramping is observed at all the stages examined. We further show that ramping dopamine signals can be driven by a dynamic stimulus that indicates a gradual approach to a reward. We provide a unified computational understanding of rapid phasic and slowly ramping dopamine signals: dopamine neurons perform a derivative-like computation over values on a moment-by-moment basis.

show abstract

“…These ramping signals could not be interpreted as phasic activity elicited by prediction error or as tonic background activity, given correspondence to a specific, prolonged epoch of goal pursuit and resolution. This signal has been proposed to be “quasi-tonic” (Lloyd and Dayan, 2015): like tonic DA, ramping DA might operate via extrasynaptic, background firing, but this remains to be comprehensively characterized. In contrast, phasic DA primarily operates via synaptic release (Floresco et al, 2003).…”

mentioning

confidence: 99%

“…This ramping dynamic has been observed in rodents traversing a maze towards a reward but also while anticipating a highly-uncertain reward (i.e., 50% reward probability (Fiorillo et al, 2003; Gershman, 2014)) suggested that ramping may reflect averaged activity elicited by reward prediction errors. Evidence from computational models suggests that DA ramps may feasibly support multiple functions: (1) resolution of action timing uncertainty; (2) increasing gain control for a chosen action; and (3) a discounted function of vigor (Lloyd and Dayan, 2015). Importantly, these functions are proposed to be non-mutually exclusive: ramping activity may reflect multiple functions at any given time.…”

mentioning

confidence: 99%

Reward Anticipation Dynamics during Cognitive Control and Episodic Encoding: Implications for Dopamine

Chiew

Stanek

Adcock

2016

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Dopamine (DA) modulatory activity critically supports motivated behavior. This modulation operates at multiple timescales, but the functional roles of these distinct dynamics on cognition are still being characterized. Reward processing has been robustly linked to DA activity; thus, examining behavioral effects of reward anticipation at different timing intervals, corresponding to different putative dopaminergic dynamics, may help in characterizing the functional role of these dynamics. Towards this end, we present two research studies investigating reward motivation effects on cognitive control and episodic memory, converging in their manipulation of rapid vs. multi-second reward anticipation (consistent with timing profiles of phasic vs. ramping DA, respectively) on performance. Under prolonged reward anticipation, both control and memory performances were enhanced, specifically when combined with other experimental factors: task-informative cues (control task) and reward uncertainty (memory task). Given observations of ramping DA under uncertainty (Fiorillo et al., 2003) and arguments that uncertainty may act as a control signal increasing environmental monitoring (Mushtaq et al., 2011), we suggest that task information and reward uncertainty can both serve as “need for control” signals that facilitate learning via enhanced monitoring, and that this activity may be supported by a ramping profile of dopaminergic activity. Observations of rapid (i.e., phasic) reward on control and memory performance can be interpreted in line with prior evidence, but review indicates that contributions of different dopaminergic timescales in these processes are not well-understood. Future experimental work to clarify these dynamics and characterize a cross-domain role for reward motivation and DA in goal-directed behavior is suggested.

show abstract

Tamping Ramping: Algorithmic, Implementational, and Computational Explanations of Phasic Dopamine Signals in the Accumbens

Cited by 47 publications

References 130 publications

Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision

Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision

A unified framework for dopamine signals across timescales

Reward Anticipation Dynamics during Cognitive Control and Episodic Encoding: Implications for Dopamine

Contact Info

Product

Resources

About