Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules

Abstract

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules.

A solution to the learning dilemma for recurrent networks of spiking neurons

“…In random e-prop the values of the weights B jk are randomly chosen and remain fixed, similar to broadcast alignment for feedforward networks Nøkland, 2016). Resulting synaptic plasticity rules (see Methods) look very similar to previously proposed plasticity rules (Gerstner et al, 2018). In particular they involve postsynaptic depolarization as one of the factors, similarly as the data-based rule in Clopath et al (2010), see section S6 in the supplement for an analysis.…”

“…In contrast to the supervised case where the learning signals depend on the deviation from an external target signal, the learning signals here are emitted when an action is taken and they express here how much this action deviates from the action mean that is currently proposed by the network. We show in Methods that reward-based e-prop yields local reward-based rules for synaptic plasticity that are in many aspects similar to ones that have previously been discussed in the literature (Gerstner et al, 2018). But those previously proposed rules estimated gradients of the policy essentially by correlating the noisy output of network neurons with rewards, which is known to be inefficient due to noisy gradient estimates.…”

“…To provide a more interpretable form of eligibility trace that fits into the standard form of local terms considered in 3-factor learning rules (Gerstner et al, 2018), one may drop the term −ψ t j β in equation (22). This approximation t ji,a of equation (22) becomes an exponential trace of the post-pre pairings accumulated within a time window as large as the adaptation adaptation time constant:…”

“…i) The dynamics of neurons in the brain is enriched by continuously ongoing updates of traces of past activity on the molecular level, for example in the form of calcium ions or activated CaMKII enzymes (Sanhueza and Lisman, 2013). These traces in particular record events where the presynaptic neuron fired before the postsynaptic neuron, which is known to induce Hebbian-like STDP (spike timing dependent plasticity) if followed by a top-down learning signal (Cassenaer and Laurent, 2012;Yagishita et al, 2014;Gerstner et al, 2018). We refer to local traces of this type as eligibility traces in our learning model.…”

Exercise during abstinence normalizes ultrastructural synaptic plasticity associated with nicotine‐seeking following extended access self‐administration

“…The elongation of synapses is likely slower because the processes needed for the integration of additional AMPARs, namely the growth of presynaptic boutons, the enlargement of the PSD, and protein synthesis (Lisman, 2017), take time. This late stage of LTP can be described as "neoHebbian," as in addition to the Hebbian requirement of synchronized activation of pre-and postsynaptic neurons, a third factor marking the salience, novelty, or reward of the stimuli and signaled through neuromodulators, such as dopamine, is also involved (Gerstner, Lehmann, Liakoni, Corneil, & Brea, 2018;Lisman, 2017). Indeed, dopamine has been reported to promote dendritic protein synthesis in the CA1 hippocampal region as well as the enlargement of dendritic spines in the NAc when paired with pre-and postsynaptic spikes (Smith, Starck, Roberts, & Schuman, 2005;Yagishita et al, 2014).…”