Few-shot learning: temporal scaling in behavioral and dopaminergic learning

Abstract

How do we learn associations in the world (e.g., between cues and rewards)? Cue-reward associative learning is controlled in the brain by mesolimbic dopamine. It is widely believed that dopamine drives such learning by conveying a reward prediction error (RPE) in accordance with temporal difference reinforcement learning (TDRL) algorithms. TDRL implementations are trial-based: learning progresses sequentially across individual cue-outcome experiences. Accordingly, a foundational assumption, often considered a mere truism, is that the more cue-reward pairings one experiences, the more one learns this association. Here, we disprove this assumption, thereby falsifying a foundational principle of trial-based learning algorithms. Specifically, when a group of head-fixed mice received ten times fewer experiences over the same total time as another, a single experience produced as much learning as ten experiences in the other group. This quantitative scaling also holds for mesolimbic dopaminergic learning, with the increase in learning rate being so high that the group with fewer experiences exhibits dopaminergic learning in as few as four cue-reward experiences and behavioral learning in nine. An algorithm implementing reward-triggered retrospective learning explains these findings. The temporal scaling and few-shot learning observed here fundamentally changes our understanding of the neural algorithms of associative learning.

Does phasic dopamine release cause policy updates?

“…By treating response‐contingent dopamine firing as a primary reward, we swim against the mainstream current: the view that such firing acts as an RPE. In contrast, our results and interpretation are consistent with a recent reformulation of the role of phasic dopamine firing in learning (Burke et al, 2023; Garr et al, 2023; Jeong et al, 2022). On that view, a burst of firing in midbrain dopamine neurons informs the animal that a significant event has occurred, triggering a retrospective search for the cause.…”

“…On that view, a burst of firing in midbrain dopamine neurons informs the animal that a significant event has occurred, triggering a retrospective search for the cause. The results of multiple new experiments carried out by the authors are interpreted to be consistent with their novel proposed function for phasic dopamine signalling and not with TDRL (Burke et al, 2023; Garr et al, 2023; Jeong et al, 2022). Moreover, they postulate that phasic dopamine signalling sums with the ‘the innate contribution to the meaning of a stimulus’ in defining causal targets for learning.…”

Does phasic dopamine release cause policy updates?

“…By treating response-contingent dopamine firing as a primary reward, we swim against the mainstream current: the view that such firing acts as an RPE. In contrast, our results and interpretation are consistent with a recent reformulation of the role of phasic dopamine firing in learning [42][43][44] . On that view, a burst of firing in midbrain dopamine neurons informs the animal that a significant event has occurred, triggering a retrospective search for the cause.…”

“…On that view, a burst of firing in midbrain dopamine neurons informs the animal that a significant event has occurred, triggering a retrospective search for the cause. The results of multiple new experiments carried out by the authors are interpreted to be consistent with their novel proposed function for phasic dopamine signaling and not with TDRL [42][43][44] . Moreover, they postulate that phasic dopamine signaling sums with the "the innate contribution to the meaning of a stimulus" in defining causal targets for learning.…”

Mesolimbic dopamine ramps reflect environmental timescales

“…In our recent work proposing that dopamine acts as a teaching signal for causal learning by representing the Adjusted Net Contingency for Causal Relations (ANCCR)(20, 34, 40), simulated ANCCR depends on the duration of a memory trace of past events (i.e., on the “eligibility trace” time constant) (illustrated in Extended Data Fig 1). Accordingly, we successfully simulated dopamine ramping dynamics assuming two conditions: a dynamic progression of cues that signal temporal proximity to reward, and a small eligibility trace time constant relative to the trial period(20).…”