Ramping activity in midbrain dopamine neurons signifies the use of a cognitive map

Abstract

Journeys to novel and familiar destinations employ different navigational strategies. The 1 first drive to a new restaurant relies on map-based planning, but after repeated trips the 2 drive is automatic and guided by local environmental cues 1,2 . Ventral striatal dopamine rises 3 during navigation toward goals and reflects the spatial proximity and value of goals 3 , but the 4 impact of experience, the neural mechanisms, and the functional significance of dopamine 5 ramps are unknown 4,5 . Here, we used fiber photometry [6][7][8] to record the evolution of activity 6 in midbrain dopamine neurons as mice learned a variety of reward-seeking tasks, starting 7 recordings before training had commenced and continuing daily for weeks. When mice 8 navigated through space toward a goal, robust ramping activity in dopamine neurons 9 appeared immediately -after the first rewarded trial on the first training day in completely 10 naïve animals. In this task spatial cues were available to guide behavior, and although ramps 11 were strong at first, they gradually faded away as training progressed. If instead mice 12 learned to run a fixed distance on a stationary wheel for reward, a task that required an 13 internal model of progress toward the goal, strong dopamine ramps persisted indefinitely. 14 In a passive task in which a visible cue and reward moved together toward the mouse, ramps 15 appeared and then faded over several days, but in an otherwise identical task with a 16 stationary cue and reward ramps never appeared. Our findings provide strong evidence that 17 ramping activity in midbrain dopamine neurons reflects the use of a cognitive map 9,10 -an 18 internal model of the distance already covered and the remaining distance until the goal is 19 reached. We hypothesize that dopamine ramps may be used to reinforce locations on the way 20 to newly-discovered rewards in order to build a graded ventral striatal value landscape for 21 guiding routine spatial behavior.The decision to continue pursuing a goal or abandon the quest depends on how much progress has 1 been made, how much remains to be done, and the value of the goal. For example, a climber will 2 be more deterred by rain at a mountain's base than near the summit, and will be more reluctant to 3 abandon a prized peak than a training hill. Information about progress toward goals and their value 4 is essential for adaptively balancing time and energy between activities, and commitment to goals 5 and the vigour of goal-directed actions are both regulated by goal progress and value 11-16 . 6 7Ventral striatal dopamine (DA) progressively rises as rodents navigate toward spatially distant 8 rewards 3 , a surprising recent finding that was not anticipated by temporal difference learning 9 models of DA function 4 but which has broadened our understanding of the role of ventral striatal 10 DA in sustaining and invigorating goal-directed behavior 17 . DA ramps reflect the value and 11 proximity of goals, scaling by the value of the reward and stretching or compressing in ...

Tonic activity in lateral habenula neurons promotes disengagement from reward-seeking behavior

“…Animals transition between directed pursuit of rewards and exploratory or quiescent behavioral states on a timescale of minutes to hours (Ferster and Skinner, 1957;Cohen et al, 2007;Flavell et al, 2013;Hills et al, 2015;Stern et al, 2017;Ebitz et al, 2018;Marques et al, 2020). Factors that influence the persistence of reward-seeking behavioral states include current and predicted homeostatic need (Aponte et al, 2011;Chen et al, 2015), reward proximity (Howe et al, 2013;McGinty et al, 2013;Westbrook and Frank, 2018;Guru et al, 2020), the history of action successes and failures (Vroom, 1964;Charnov, 1976;Ullsperger and von Cramon, 2003;Ebitz et al, 2019), opportunity costs (Niv et al, 2007;Kurzban et al, 2013;Boureau et al, 2015), and environmental threats (Lecca et al, 2017;Alhadeff et al, 2018).…”

“…Indeed, there is anatomical and behavioral evidence that supports this idea. Midbrain dopamine (DA) neurons play an essential role in supporting sustained goal-directed behavior (Salamone and Correa, 2012;Dolan and Dayan, 2013;Howe et al, 2013;Guru et al, 2020), and the LHb inhibits DA neural activity via the GABAergic rostromedial tegmental nucleus (RMTg) (Christoph et al, 1986;Ji and Shepard, 2007;Matsumoto and Hikosaka, 2007;Hong et al, 2011). LHb lesions reduce reward omission dips in DA neural activity (Tian and Uchida, 2015), LHb stimulation reduces the number of actions that animals are willing to perform for rewards (Proulx et al, 2018), and LHb stimulation reduces entries to spatial locations where stimulation is delivered (Stamatakis and Stuber, 2012).…”

Inferring spikes from calcium imaging in dopamine neurons

“…In addition to the phasic modulation of DA activity in response to reward or reward-predicting cues, there are an increasing number of reports suggesting that activity may "ramp" as animals approach a reward in space (22,(52)(53)(54)(55)(56) . Most of these studies were based on calcium imaging (22,56) , or on measures of downstream DA release (52,53,55) , but more recently evidence has emerged from in vivo electrophysiology of ramps in the tonic firing of VTA DA neurons during reward approach (54) .…”

Reduced Successor Representation Potentially Interferes with Cessation of Habitual Reward-Seeking

“…Successor representation and value update based on it have been suggested to be implemented in the prefrontal/hippocampus-dorsomedial/ventral striatum circuits (Garvert et al , 2017; Russek et al , 2017; Stachenfeld et al , 2017), while circuits including dorsolateral striatum might implement habitual or model-free behavior through “punctate” (i.e., individual) representation of states or actions. Sustained DA response to predictable reward, possibly related to state representation The original experiments that led to the proposal of representation of RPE by DA (Montague et al , 1996; Schultz et al , 1997) have shown that DA response to reward disappears after monkeys repeatedly experienced the stimulus(-action)-reward association and the reward presumably became predictable for them. However, sustained, and often ramping, dopamine signals to/towards (apparently) predictable reward has been widely observed in recent years (Howe et al , 2013; Collins et al , 2016; Hamid et al , 2016; Hamid et al , 2019; Kim et al , 2019; Mohebi et al , 2019; Guru et al , 2020; Sarno et al , 2020). There are a number of possible accounts for such sustained DA signals, positing that they represent RPE (Gershman, 2014; Morita & Kato, 2014; Kato & Morita, 2016; Kim et al , 2019; Mikhael et al , 2019; Song & Lee, 2020) or something different from RPE (Howe et al , 2013; Hamid et al , 2016; Hamid et al , 2019; Mohebi et al , 2019; Guru et al , 2020; Sarno et al , 2020) or both (Lloyd & Dayan, 2015; Collins et al , 2016).…”

“…However, sustained, and often ramping, dopamine signals to/towards (apparently) predictable reward has been widely observed in recent years (Howe et al , 2013; Collins et al , 2016; Hamid et al , 2016; Hamid et al , 2019; Kim et al , 2019; Mohebi et al , 2019; Guru et al , 2020; Sarno et al , 2020). There are a number of possible accounts for such sustained DA signals, positing that they represent RPE (Gershman, 2014; Morita & Kato, 2014; Kato & Morita, 2016; Kim et al , 2019; Mikhael et al , 2019; Song & Lee, 2020) or something different from RPE (Howe et al , 2013; Hamid et al , 2016; Hamid et al , 2019; Mohebi et al , 2019; Guru et al , 2020; Sarno et al , 2020) or both (Lloyd & Dayan, 2015; Collins et al , 2016). Of particular interest to our present work, one hypothesis (Gershman, 2014) suggests that sustained (ramping) DA signals might represent sustained RPE generated due to imperfect approximation of value function in the system using representation of states by low-dimensional features.…”