Dopamine selectively remediates ‘model-based’ reward learning: a computational approach

Abstract

Patients with loss of dopamine due to Parkinson's disease are impaired at learning from reward. However, it remains unknown precisely which aspect of learning is impaired. In particular, learning from reward, or reinforcement learning, can be driven by two distinct computational processes. One involves habitual stamping-in of stimulus-response associations, hypothesized to arise computationally from 'model-free' learning. The other, 'model-based' learning, involves learning a model of the world that is believed to support goal-directed behaviour. Much work has pointed to a role for dopamine in model-free learning. But recent work suggests model-based learning may also involve dopamine modulation, raising the possibility that model-based learning may contribute to the learning impairment in Parkinson's disease. To directly test this, we used a two-step reward-learning task which dissociates model-free versus model-based learning. We evaluated learning in patients with Parkinson's disease tested ON versus OFF their dopamine replacement medication and in healthy controls. Surprisingly, we found no effect of disease or medication on model-free learning. Instead, we found that patients tested OFF medication showed a marked impairment in model-based learning, and that this impairment was remediated by dopaminergic medication. Moreover, model-based learning was positively correlated with a separate measure of working memory performance, raising the possibility of common neural substrates. Our results suggest that some learning deficits in Parkinson's disease may be related to an inability to pursue reward based on complete representations of the environment.

Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions

“…What we show here is that in the presence of an external reward, the recruitment of a model-based learning strategy is not an exception but rather a central feature of VTA-DA teaching signals. This is consistent with recent studies showing that treatments (pharmacological or dietary restrictions) that globally increase or decrease DA function promote or impair, respectively, model-based processes in humans [35–37]. …”

The costs and benefits of brain dopamine for cognitive control

“…The potentiating effects of prefrontal dopamine on the distractor resistance of current working memory representations in prefrontal cortex might also contribute to the enhancing effects of dopaminergic medication in Parkinson's disease on goal‐directed (over habitual) control of behavior, which relies on the ability to maintain online an explicit representation of the (route to the) outcome of behavior. Indeed, infusions of dopamine into the prefrontal cortex restored outcome sensitivity in experimental animals, putatively by engaging attentional/working memory processes.…”

“…Critically, individual differences in the dopaminergic drug effects on distractor resistance were found to correlate with drug effects on delay period signal in the prefrontal cortex, as well as on functional connectivity between the prefrontal cortex and stimulus-specific regions in the posterior visual association cortex. 43 The potentiating effects of prefrontal dopamine on the distractor resistance of current working memory representations in prefrontal cortex might also contribute to the enhancing effects of dopaminergic medication in Parkinson's disease on goal-directed (over habitual) control of behavior, 47,48 which relies on the ability to maintain online an explicit representation of the (route to the) outcome of behavior. Indeed, infusions of dopamine into the prefrontal cortex restored outcome sensitivity in experimental animals, putatively by engaging attentional/working memory processes 49 .…”

When Does Model-Based Control Pay Off?

“…This task has proven to be a useful and popular tool to characterize the neural [8, 10–18], behavioral [19–31] and clinical [32–35] implications of dual-process models within the reinforcement learning framework. However, in this paper we argue that the two-step task does not induce a trade-off between accuracy and demand: Our simulations show that the “deliberative” strategy does not increase performance accuracy on the task.…”