An association between prediction errors and risk-seeking: Theory and behavioral evidence

Abstract

Reward prediction errors (RPEs) and risk preferences have two things in common: both can shape decision making behavior, and both are commonly associated with dopamine. RPEs drive value learning and are thought to be represented in the phasic release of striatal dopamine. Risk preferences bias choices towards or away from uncertainty; they can be manipulated with drugs that target the dopaminergic system. Based on the common neural substrate, we hypothesize that RPEs and risk preferences are linked on the level of behavior as well. Here, we develop this hypothesis theoretically and test it empirically. First, we apply a recent theory of learning in the basal ganglia to predict how RPEs influence risk preferences. We find that positive RPEs should cause increased risk-seeking, while negative RPEs should cause risk-aversion. We then test our behavioral predictions using a novel bandit task in which value and risk vary independently across options. Critically, conditions are included where options vary in risk but are matched for value. We find that our prediction was correct: participants become more risk-seeking if choices are preceded by positive RPEs, and more risk-averse if choices are preceded by negative RPEs. These findings cannot be explained by other known effects, such as nonlinear utility curves or dynamic learning rates.

Dopamine increases risky choice while D2 blockade shortens decision time

“…We hypothesise that levodopa increases risk-taking behaviour via overstimulation at both D1 and D2 receptor level, while a single low dose of haloperidol, as previously reported (Frank and O'Reilly 2006), may block D2 receptors pre-and post-synaptically and may paradoxically lead to higher striatal dopamine acting on remaining striatal D1 receptors, causing speedier decision without influencing risk tolerance. These effects could also fit with a recently proposed computational model of the basal ganglia (Moeller and Bogacz 2019;Moeller et al 2021). Furthermore, our data suggest that the actual dopaminergic drug effect may be dependent on the individual's baseline dopamine state, which may influence our therapeutic decision as clinicians in the future.…”

“…For example, pathological gambling is an impulse control disorder that involves increased risk-taking and is linked to hyperdopaminergic states (Pine et al 2010;Sinha et al 2013;Voon et al 2010;du Hoffmann and Nicola 2014). In line with this, it has been proposed that dopamine controls the effect of risk on choice (Moeller et al 2021). According to this theory, increasing dopamine levels should lead to risky decision-making.…”

“…According to a recently proposed model of the basal ganglia (Opponent Actor Learning model), the value of an option is computed by subtracting its expected losses, encoded in the No-Go pathway, from its expected gains, encoded in the Go pathway (Collins and Frank 2014 ). Dopamine may control the relative contribution of these two pathways (Mikhael and Bogacz 2016 ; Moeller et al 2021 ): …”

Uncertainty–guided learning with scaled prediction errors in the basal ganglia

“…In Eqs 5 and 6 , G and N denote the synaptic inputs in the direct and indirect pathway respectively [ 22 ], and λ is a coefficient determining the accuracy with which the standard deviation can be encoded (as explained below). Term –1 in Eq 6 in necessary for the neural implementation of scaled prediction error computation (as will be seen later).…”

“…However, it has been recently demonstrated that even a brief, burst–like activation of dopaminergic neurons changes the activity levels of striatal neurons [ 39 ]. Additionally, it has been shown that reward prediction errors modulate the tendency to make risky choices [ 22 ], and risk attitudes are known to depend on the balance between the activities in direct and indirect pathways [ 40 , 41 ]. In this paper, we demonstrated that a more realistic assumption, that the dopamine signal encoding prediction error also changes the activity levels in striatum, enables scaling of prediction errors by uncertainty.…”