Does phasic dopamine release cause policy updates?

Abstract

Phasic dopamine activity is believed to both encode reward-prediction errors (RPEs) and to cause the adaptations that these errors engender. If so, a rat working for optogenetic stimulation of dopamine neurons will repeatedly update its policy and/or action values, thus iteratively increasing its work rate. Here, we challenge this view by demonstrating stable, non-maximal work rates in the face of repeated optogenetic stimulation of midbrain dopamine neurons. Furthermore, we show that rats learn to discriminate between world states distinguished only by their history of dopamine activation. Comparison of these results to reinforcement learning simulations suggests that the induced dopamine transients acted more as rewards than RPEs. However, pursuit of dopaminergic stimulation drifted upwards over a time scale of days and weeks, despite its stability within trials. To reconcile the results with prior findings, we consider multiple roles for dopamine signaling.

Reward timescale controls the rate of behavioral and dopaminergic learning

“…In addition, we establish that dopaminergic learning exhibits temporal scaling, whereas the asymptotic response does not (Fig 2 ), thereby offering a significant new constraint for dopamine-mediated learning models. While prior work has provided accumulating evidence that mesolimbic dopamine signals do not function strictly as a TDRL reward prediction error signal 3,52,[71][72][73][74][75][76] , the current results call into question the broader trial-based reinforcement learning framework used to understand dopamine and learning. While some prior models do explain the quantitative scaling of behavioral learning, these models do not yet explain dopaminergic dynamics 35,36,77,78 .…”

Mesostriatal dopamine is sensitive to changes in specific cue-reward contingencies

“…The use of outcome-specific contingency degradation combined with single-unit DA neuron recordings will be a productive line of research going forward. The present work adds to a growing complement of studies that require us to expand our understanding of DA's role in learning (12,14,25,(34)(35)(36)(37)(38).…”

Rewarding Value or Prediction Error: Settling the debate over the role of dopamine in reward learning

“…The dominance of each population, however, may be determined by the context in which it is tested. For example, stimulation in the context of classical conditioning drives a prediction error signal (e.g., Steinberg et al, 2013; Keiflin et al, 2018; Sharpe et al, 2017), whereas stimulation in an instrumental setting delivers rewarding value (e.g, Carter et al, 2022; Witten et al, 2010). While there may be some rewarding property embedded in the general DA signal, an error mechanism would parsimoniously account for the reinforcement property of VTA DA activation without having to appeal to the induction of rewarding value.…”

“…It is incontrovertible that VTA DA stimulation acts as a reinforcing signal for actions and states (e.g., Olds & Milner, 1954; Wise, 1978; Schultz et al, 1997). Animals readily self-stimulate for, and frequent places where, electrical or optogenetic activation of VTA DA neurons occurred, and disrupting this activity prevents learning or reduces established reward-seeking responses (e.g., Carter et al, 2022; Corbett & Wise, 1980; Crow, 1972; Ilango et al, 2014; Pascoli et al, 2015; Phillips & Fibiger, 1973; Millard et al, 2022; Witten et al, 2010). An intuitive interpretation of these findings is that VTA DA activity constitutes an appetitive event that is intrinsically rewarding.…”