Dopaminergic Control of Motivation and Reinforcement Learning: A Closed-Circuit Account for Reward-Oriented Behavior

Abstract

Humans and animals take actions quickly when they expect that the actions lead to reward, reflecting their motivation. Injection of dopamine receptor antagonists into the striatum has been shown to slow such reward-seeking behavior, suggesting that dopamine is involved in the control of motivational processes. Meanwhile, neurophysiological studies have revealed that phasic response of dopamine neurons appears to represent reward prediction error, indicating that dopamine plays central roles in reinforcement learning. However, previous attempts to elucidate the mechanisms of these dopaminergic controls have not fully explained how the motivational and learning aspects are related and whether they can be understood by the way the activity of dopamine neurons itself is controlled by their upstream circuitries. To address this issue, we constructed a closed-circuit model of the corticobasal ganglia system based on recent findings regarding intracortical and corticostriatal circuit architectures. Simulations show that the model could reproduce the observed distinct motivational effects of D 1 -and D 2 -type dopamine receptor antagonists. Simultaneously, our model successfully explains the dopaminergic representation of reward prediction error as observed in behaving animals during learning tasks and could also explain distinct choice biases induced by optogenetic stimulation of the D 1 and D 2 receptor-expressing striatal neurons. These results indicate that the suggested roles of dopamine in motivational control and reinforcement learning can be understood in a unified manner through a notion that the indirect pathway of the basal ganglia represents the value of states/actions at a previous time point, an empirically driven key assumption of our model. IntroductionDopamine has been suggested to control motivation and rewardseeking behavior (Robbins and Everitt, 1996;Berridge and Robinson, 1998;Dayan and Balleine, 2002; McClure et al., 2003;. As a direct evidence, application of dopamine receptor antagonists in the striatum has been shown to slow the subject's behavior (Salamone and Correa, 2002; Nakamura and Hikosaka, 2006), with distinct effects observed for the antagonists of D1-and D2-type dopamine receptors (D1Rs and D2Rs), which are expressed in distinct populations of striatal medium spiny neurons (MSNs) projecting to the "direct" (dMSNs) and "indirect" (iMSNs) pathways of the basal ganglia, respectively (Gerfen and Surmeier, 2011). Clarifying mechanisms of such pharmacological effects is likely to help elucidating the exact roles of dopamine in motivational control, and neural circuit modeling has been used as one of the powerful approaches (Frank et al., 2004;. However, these models are still not self-contained in a sense that responses of either the dopamine neurons or their hypothesized upstream globus pallidus (GP) neurons were presumed rather than explained by inputs from the rest part of the circuit.Along with the suggested roles in motivational control, dopamine has also been suggested to be centrally...

Differential cortical activation of the striatal direct and indirect pathway cells: reconciling the anatomical and optogenetic results by using a computational method

“…the PT inputs to dMSNs have been proposed to cause dendritic activation that works as a marker for the induction of dopaminedependent synaptic plasticity (Morita et al 2013). Here I extend the latter proposal on the basis of results of the present study.…”

“…12. Functional implications of the results of the present study, in relation to the hypothetical mechanism of reinforcement leaning and action selection in the corticobasal ganglia circuit named the corticostriatal temporal difference (TD) hypothesis (Morita et al 2012(Morita et al , 2013. A: hypothetical mechanism of the computation of reward prediction error (RPE) of action values in the midbrain dopamine neurons (DA cells), along with action execution and termination in the corticobasal ganglia-thalamocortical loop, according to the corticostriatal TD hypothesis.…”

“…Since dMSNs and iMSNs would potentially cause net positive and negative impacts on the activity of midbrain dopamine neurons via the output nuclei of the basal ganglia, respectively, the difference of the values of the current and previous actions/ states, which is the core of the reward prediction error (RPE) defined in the reinforcement learning theory (called the TD error; Sutton and Barto 1998), can be computed in the dopamine neurons, providing a possible mechanistic account for the suggested dopaminergic representation of RPE (Montague et al 1996;Schultz et al 1997). Along with being involved in the computation of RPE in such a way, dMSNs and iMSNs are hypothesized to also engage in execution (initiation) and termination of action through disinhibition and reinhibition of the thalamocortical feedback inputs, respectively, as has been suggested (Nambu 2008), with the layer 1-preferring (Kuramoto et al 2009) thalamocortical inputs presumably targeting the extensively branched tuft dendrites of PT cells and triggering/boosting PT cells' output to the pyramidal tract (or other cortical areas) to initiate an action (or action plan) (Morita et al 2012(Morita et al , 2013. A critical assumption of this corticostriatal TD hypothesis is, as described above, the preferential activation of dMSNs and iMSNs by IT cells and PT cells, respectively.…”

“…Recently, a hypothesis on the specific roles of the corticostriatal system in these cognitive functions was proposed (Morita et al 2012(Morita et al , 2013. According to this hypothesis, referred to as the corticostriatal temporal difference (TD) hypothesis (Fig.…”

“…What are, then, the functional roles of IT¡iMSN and PT¡dMSN inputs, which presumably have rather strong impacts at the beginning but then decay with time? In fact, the corticostriatal TD hypothesis (Morita et al 2012(Morita et al , 2013 posits that not only the presumably major IT¡dMSN and PT¡iMSN pathways but also the IT¡iMSN and PT¡dMSN pathways have important functional roles. Specifically, the IT¡iMSN pathway has been suggested to be possibly involved in the temporal discount of future rewards (Morita et al 2012), and Fig.…”