A temporal difference account of avoidance learning

Moutoussis, Michael; Bentall, Richard P.; Williams, Jonathan; Dayan, Peter

doi:10.1080/09548980802192784

Cited by 81 publications

(75 citation statements)

References 76 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two-factor theories of active avoidance behavior (34)(35)(36) suggest that the transition from an unsafe to a safe state is coded similarly to a reward, an idea central to various notions about vigor and learning in avoidance (13). The lack of an effect of levodopa in the go to avoid losing condition is troubling for this account.…”

Section: Discussionmentioning

confidence: 99%

Action controls dopaminergic enhancement of reward representations

Guitart-Masip

Chowdhury

Sharot

et al. 2012

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

108

144

View full text Add to dashboard Cite

Dopamine is widely observed to signal anticipation of future rewards and thus thought to be a key contributor to affectively charged decision making. However, the experiments supporting this view have not dissociated rewards from the actions that lead to, or are occasioned by, them. Here, we manipulated dopamine pharmacologically and examined the effect on a task that explicitly dissociates action and reward value. We show that dopamine enhanced the neural representation of rewarding actions, without significantly affecting the representation of reward value as such. Thus, increasing dopamine levels with levodopa selectively boosted striatal and substantia nigra/ventral tegmental representations associated with actions leading to reward, but not with actions leading to the avoidance of punishment. These findings highlight a key role for dopamine in the generation of appetitively motivated actions.functional MRI | striatum | decision-making S ubstantial evidence indicates that the neuromodulator dopamine plays at least two roles in behavioral guidance. One is to signal prediction errors regarding expected reward value (1-3), prediction errors that are also used by the striatum to guide selection of appropriate actions (4-6). A second, less emphasized, role for dopamine is to invigorate actions associated with reward (7,8). Thus, dopamine depletion results in decreased motor activity and decreased motivated behavior (9, 10) along with decreased vigor or motivation to work for rewards in demanding reinforcement schedules (7,11). These joint roles in reward prediction and motivational control can create situations that result in conflict, such as what happens when reward is attainable solely by not acting (no-go).It has been suggest that the exact opposite may be true of serotonin. That is, although the nature of any opponency between appetitive and aversive systems remains the subject of much debate (12-14), there are suggestions that serotonin acts as a mirror to dopamine and is associated with behavioral inhibition in aversive contexts (13-17). These joint roles would lead to conflict when punishment can be avoided only by acting (go).Such conflicting cases have been a focal point of recent studies where the critical manipulation involved an orthogonalization of action requirements and outcome valence (15,18). In the latter study we highlighted anticipatory representations in the striatum and substantia nigra/ventral tegmental area (SN/VTA) that reflected a dominance of action representation over an expected dominance of reward value representations, thereby hinting at a specific role for dopamine in motivation for action (18).Here, we tested the contributions of dopamine and serotonin to action and valence representations, with a specific focus on the areas we had previously highlighted, namely the striatum and SN/ VTA. First, we trained participants on a balanced 2 (reward/ punishment) × 2 (go/no-go) task that orthogonalized action and valence (18). When performance reached 95% correct choices in all conditions...

show abstract

Section: Discussionmentioning

confidence: 99%

Action controls dopaminergic enhancement of reward representations

Guitart-Masip

Chowdhury

Sharot

et al. 2012

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

108

144

View full text Add to dashboard Cite

show abstract

“…However, it is unlikely to be solely responsible, because the increased dopamine release is often large and can occur during stimulus presentation (29). Importantly, this observation can help explain why dopamine receptor antagonists interfere with avoidance learning, where the rewarding role of the offset of an aversive stimulus drives behavior (30).…”

Section: Phasic Excitation Of Dopamine Neurons At the Termination Of mentioning

confidence: 99%

Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli

Brischoux

Chakraborty

Brierley

et al. 2009

Proc. Natl. Acad. Sci. U.S.A.

687

670

View full text Add to dashboard Cite

Midbrain dopamine neurons play central roles in reward processing. It is widely assumed that all dopamine neurons encode the same information. Some evidence, however, suggests functional differences between subgroups of dopamine neurons, particularly with respect to processing nonrewarding, aversive stimuli. To directly test this possibility, we recorded from and juxtacellularly labeled individual ventral tegmental area (VTA) dopamine neurons in anesthetized rats so that we could link precise anatomical position and neurochemical identity with coding for noxious stimuli. Here, we show that dopamine neurons in the dorsal VTA are inhibited by noxious footshocks, consistent with their role in reward processing. In contrast, we find that dopamine neurons in the ventral VTA are phasically excited by footshocks. This observation can explain a number of previously confusing findings that suggested a role for dopamine in processing both rewarding and aversive events. Taken together, our results indicate that there are 2 functionally and anatomically distinct VTA dopamine systems.aversive ͉ midbrain ͉ reward ͉ salient ͉ stress

show abstract

“…A related model was later proposed by Moutoussis, Bentall, Williams, and Dayan (2008). Their model, however, was based on advantage learning (Baird, 1993;Dayan & Bald leine, 2002), whereas the model presented here is based on the standard actor-critic (Barto, 1995;Barto et al, 1983;Sutton & Barto, 1998).…”

mentioning

confidence: 99%