Dissociating neural learning signals in human sign- and goal-trackers

Abstract

Individuals differ in how they learn from experience. In Pavlovian conditioning paradigms, where cues predict reinforcer delivery at a different goal location, some animals-so-called signtrackers-come to approach the cue, whereas others, called goal-trackers, approach the goal. In sign-trackers, model-free phasic dopaminergic reward prediction errors underlie learning, which renders stimuli 'wanted'. Goal-trackers do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in 128 male humans using eye-tracking, pupillometry and fMRI informed by computational models of sign-and goal-tracking. We show that sign-trackers exhibit a neural reward prediction error signal that is not detectable in goal-trackers. Model-free value only guides gaze and pupil dilation in sign-trackers. Goal-trackers instead exhibit a stronger model-based neural state prediction error signal. This model-based construct determines gaze and pupil dilation more in goal-trackers. Main Learning from reinforcements involves multiple processes with distinct computational, neural and behavioral signatures. Consider a simple classical Pavlovian conditioning paradigm, where a cue (conditioned stimulus, CS) becomes predictive of a reward (unconditioned stimulus, US) by being repeatedly presented before the US. In so-called model-free reinforcement learning, learning occurs via reward prediction errors (RPE 1) which quantify the difference between the value of the US that actually arrives and a current prediction of that value made on the basis of the CS. Integration of the experienced RPEs allows the predictions made by the CS, called a 'cached value' to become accurate. An alternative way of learning involves building a model which has two components: a 'transition structure', which captures the probability that one stimulus is followed by another, and a 'reward structure', which captures the value associated with each particular stimulus. Learning the transition structure 2,3 can occur by integration of a different sort of so-called state prediction errors (SPE 4) which quantify the difference between the stimulus that actually occurs and the probability of this event that was estimated on the basis of the previous stimulus. This type of learning does not conflate transitions and rewards, and is hence more flexible when one of these changes. However, it

Understanding Addiction Using Animal Models

“…These findings suggest that in contrast to the HR/LR model, the main strength of the ST/GT model is elucidating the neurobiological mechanisms associated with individual variation in relapse propensity. This model also has translational validity, as work in humans has shown sign- and goal-tracking behavior (Garofalo and di Pellegrino, 2015; Joyner et al, 2018; Schad et al, 2019), though linking a specific conditioned response in humans with addiction-related behaviors has yet to be explored.…”

Fronto-striatal structures related with model-based control as an endophenotype for obsessive–compulsive disorder

“…This is in line with previous studies suggesting a prominent role for these structure in MB control. For instance, previous functional imaging studies found neural signature of MB learning signal in the dlPFC and the intraparietal sulcus in HCs 33 , 34 . Another study reported impaired MB control after transient disruption of the right dlPFC via theta burst transcranial magnetic stimulation 31 .…”

Vulnerability to relapse under stress: insights from affective neuroscience

“…Despite the potential contribution that sign-versus goaltracking propensities could have for identifying risk profiles for substance addiction, the development of paradigms measuring these individual differences in humans is just beginning [141][142][143]. A consistent amount of research has shown that -similar to animals -humans can also be attracted to the Pavlovian cue themselves and, typically, human attention is very rapidly oriented toward Pavlovian cues [24,116].…”

“…A consistent amount of research has shown that -similar to animals -humans can also be attracted to the Pavlovian cue themselves and, typically, human attention is very rapidly oriented toward Pavlovian cues [24,116]. However, to date, only a few studies have systematically investigated the individual differences in human sign-versus goal-tracking behaviours [142,143]. These new translational studies are developing paradigms that measure conditioned approaches toward the Pavlovian cue or the location of the reward delivery in a human population.…”