Asymmetric and adaptive reward coding via normalized reinforcement learning

Abstract

Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.

Craving money? Evidence from the laboratory and the field

“…Define V (1) ∝ 1 x (1) x , and then, for t > 1V(t)∝normalθt−1V(1)+true∑k=2tnormalθt−k1x(normalκ)×true(xnormalω+x+truetrue∑u=1k−1θk−u1x(u)xtrue) where ω is a positive constant reflecting saturation in neuronal responses. In this expression for the Pavlovian learned value V ( t ), which was inspired by ( 36 ), reward value is normalized by the recent reward history ∑u=1k−1normalθk−u1xfalse(ufalse)x , to account for the well-evidenced phenomenon of “divisive normalization” ( 37 ). The basic idea is that the perception of a stimulus depends not on its absolute intensity but rather on its intensity relative to what the agent expects (which is assumed to depend on the recent outcome history).…”

“…where ω is a positive constant reflecting saturation in neuronal responses. In this expression for the Pavlovian learned value V(t), which was inspired by (36), reward value is normalized by the recent reward history…”

Reward prediction error neurons implement an efficient code for reward

“…Specifically, we extended the learning rule from DRL [15] to account not only for the relationship between asymmetry and threshold but also for the three other tuning properties predicted by the efficient code. Besides DRL, there are two alternative explanations of the data by Eshel et al [16]: the Laplace code [30] and normalized reinforcement learning [31]. In the Laplace code, TD-learning neurons with different parameters are used to encode the timing and a whole distribution of rewards in the future by representing an analogue of the Laplace transform in the neural responses.…”

“…In their formulation, they subtract the expected response from each neuron instead of changing the gain (as we predict and observe). Normalized reinforcement learning [31] proposes that RPENs perform divisive normalization with different half saturation constants. This yields sigmoid (Naka-Rushton [32]) neurons with different thresholds, and the asymmetry around those thresholds is explained again by cutting the sigmoids at different heights.…”

Dopamine neurons encode a multidimensional probabilistic map of future reward

“…We used an efficient population coding framework [22][23][24] In order to optimize this efficient population code online, we generalize the distributional learning rules (Figure 5A 6 ) to the time domain, considering multiple channels with different relative scaling for over and underestimation of reward times, that generate a diversity of learnt reward time scales (Figure 5E), Importantly, these parameters converge to the efficient code that optimally adapts to the statistics of expected reward times in the environment (see Methods).…”