Adaptive properties of differential learning rates for positive and negative outcomes

Abstract

The concept of the reward prediction error-the difference between reward obtained and reward predicted-continues to be a focal point for much theoretical and experimental work in psychology, cognitive science, and neuroscience. Models that rely on reward prediction errors typically assume a single learning rate for positive and negative prediction errors. However, behavioral data indicate that better-than-expected and worse-than-expected outcomes often do not have symmetric impacts on learning and decision-making. Furthermore, distinct circuits within cortico-striatal loops appear to support learning from positive and negative prediction errors, respectively. Such differential learning rates would be expected to lead to biased reward predictions and therefore suboptimal choice performance. Contrary to this intuition, we show that on static "bandit" choice tasks, differential learning rates can be adaptive. This occurs because asymmetric learning enables a better separation of learned reward probabilities. We show analytically how the optimal learning rate asymmetry depends on the reward distribution and implement a biologically plausible algorithm that adapts the balance of positive and negative learning rates from experience. These results suggest specific adaptive advantages for separate, differential learning rates in simple reinforcement learning settings and provide a novel, normative perspective on the interpretation of associated neural data.

Do learning rates adapt to the distribution of rewards?

“…The effect of reward rate was primarily driven by the results of Experiment 1; when examined individually, only Experiment 1 showed a significant effect of reward rate [F (1, 37) = 5.73, p < 0.05]. Importantly, we found no interaction between prediction error type and reward rate (p = 0.12), disconfirming the predictions of Cazé and van der Meer (2013). We also found no effect of experiment (p = 0.94), indicating that small variations in the reward probabilities do not exert a significant effect on the learning rate asymmetry.…”

“…These updates are similar to the meta-learning algorithm proposed by Cazé and van der Meer (2013), which estimates the optimal learning rates. Intuitively, these updates will cause η − to increase on high-reward rate blocks and to decrease on low-reward rate blocks, while the opposite pattern will obtain for η + .…”

“…In contrast, Cazé and van der Meer (2013) have recently hypothesized that the asymmetry may dynamically adapt to the distribution of rewards across options. Their hypothesis is based on a normative argument: Asymmetric learning rates can enable an agent to better discriminate reward probabilities, and thereby earn more reward.…”

“…Importantly, the optimal asymmetry depends on the average reward rate, such that the learning rate for negative prediction errors should be higher than the learning rate for positive prediction errors when the average reward rate is high, and this relationship should reverse when the reward rate is low. Cazé and van der Meer (2013) proposed a metalearning algorithm that automatically adapts the asymmetry based on the reward history, and they showed in simulations that this algorithm leads to superior performance compared to an algorithm with fixed learning rates.…”

“…We then fit several different reinforcement learning models and performed formal model comparison. These models include standard RL models (Daw, O'Doherty, Dayan, Seymour, & Dolan, 2006;Sutton & Barto, 1998), as well as models with asymmetric learning rates (Daw, Kakade, & Dayan, 2002;Frank, Moustafa, Haughey, Curran, & Hutchison, 2007;Frank, Doll, Oas-Terpstra, & Moreno, 2009;Niv, Edlund, Dayan, & O'Doherty, 2012) and variants of the meta-learning model proposed by Cazé and van der Meer (2013). Taken together, these models cover a range of assumptions concerning learning rates that have been proposed in the recent RL literature.…”