A distributional code for value in dopamine-based reinforcement learning

Abstract

Since its introduction, the reward prediction error (RPE) theory of dopamine has explained a wealth of empirical phenomena, providing a unifying framework for understanding the representation of reward and value in the brain 1-3 . According to the now canonical theory, reward predictions are represented as a single scalar quantity, which supports learning about the expectation, or mean, of stochastic outcomes. In the present work, we propose a novel account of dopamine-based reinforcement learning. Inspired by recent artificial intelligence research on distributional reinforcement learning 4-6 , we hypothesized that the brain represents possible future rewards not as a single mean, but instead as a probability distribution, effectively representing multiple future outcomes simultaneously and in parallel. This idea leads immediately to a set of empirical predictions, which we tested using single-unit recordings from mouse ventral tegmental area. Our findings provide strong evidence for a neural realization of distributional reinforcement learning.The RPE theory of dopamine derives from work in the artificial intelligence (AI) field of reinforcement learning (RL) 7 . Since the link to neuroscience was first made, however, RL has made substantial advances 8,9 , revealing factors that radically enhance the effectiveness of RL algorithms 10 . In some cases, the relevant mechanisms invite comparison with neural function, suggesting new hypotheses concerning reward-based learning in the brain [11][12][13] . Here, we examine one particularly promising recent development in AI research and

Subthreshold repertoire and threshold dynamics of midbrain dopamine neuron firing in vivo

“…5K). This is in line with the recent idea of a distributed population code for dopamine neurons (Dabney et al, 2020).…”

“…Dopamine neurons signal prediction errors by transient, sub-second changes in their firing rates. A recent study implied that prediction error coding in dopamine neurons might occur in a distributed manner, thereby creating a range from "optimistic" to "pessimistic" cellular coding schemes (Dabney et al, 2020). For many years, transient cue-or reward-associated increases in firing rate -from a low tonic background frequency (about 1-8 Hz) into the beta and gamma range (15-30 Hz and 40-100 Hz, respectively) -were thought to encode positive reward prediction errors (RPEs), while short rate reductions or pauses in baseline firing were believed to encode negative RPEs, induced for instance by the omission of an expected reward.…”

Reverse-engineering Recurrent Neural Network solutions to a hierarchical inference task for mice

“…These questions are of interest to both the neuroscience and the machine learning communities. To neuroscience, RNNs are neurally-plausible mechanistic models that can serve as a good comparison with animal behavior and neural data, as well as a source of scientific hypotheses [15,16,31,5,27,10,8]. To machine learning, we build on prior work reverse engineering how RNNs solve tasks [30,28,16,15,3,20,14,19], by studying a complicated task that nevertheless has exact Bayesian baselines for comparison, and by contributing task-agnostic analysis techniques.…”

Asymmetric representation of aversive prediction errors in Pavlovian threat conditioning

“…Negative PEs in our study correspond to better-thanexpected outcomes. We note that many dopaminergic midbrain neurons encode better-than-expected outcomes in increased firing rates, and worse-than-expected outcomes in reduced firing rates, and this reduction is often less pronounced than the increase 33 , despite variability between individual neurons 29 .…”

“…As a second possible reason, biased PE encoding in individual neurons can, when integrated on the population level, afford probabilistic learning 29 . This study addressed variability of reward PE encoding bias in neurons within one region, but the same mechanism could also act across regions.…”