An anatomical substrate of credit assignment in reinforcement learning

Abstract

Learning turns experience into better decisions. A key problem in learning is credit assignment-knowing how to change parameters, such as synaptic weights deep within a neural network, in order to improve behavioral performance. Artificial intelligence owes its recent bloom largely to the error-backpropagation algorithm 1 , which estimates the contribution of every synapse to output errors and allows rapid weight adjustment. Biological systems, however, lack an obvious mechanism to backpropagate errors. Here we show, by combining high-throughput volume electron microscopy 2 and automated connectomic analysis [3][4][5] , that the synaptic architecture of songbird basal ganglia supports local credit assignment using a variant of the node perturbation algorithm proposed in a model of songbird reinforcement learning 6,7 . We find that key predictions of the model hold true: first, cortical axons that encode exploratory motor variability terminate predominantly on dendritic shafts of striatal spiny neurons, while cortical axons that encode song timing terminate almost exclusively on spines. Second, synapse pairs that share a presynaptic cortical timing axon and a postsynaptic spiny dendrite are substantially more similar in size than expected, indicating Hebbian plasticity 8,9 . Combined with numerical simulations, these findings provide strong evidence for a biologically plausible credit assignment mechanism 6 . Neural circuits that control decisions and actions are recurrently connected and involve many network layers from sensory inputs to motor output. Yet, as we learn, some mechanism specifies precisely which synapses, out of trillions, are to be modified and in what way. The backpropagation algorithm 10 is powerful because it directly calculates, based on the network architecture, the derivative of output errors with respect to every synaptic weight, providing an efficient method to update synaptic strengths. However, it remains unclear whether backpropagation or its variants are biologically implemented, or even plausible [11][12][13] . An alternative approach to implement gradient-based learning is weight-or node-perturbation 14,15 , in which the activity of a specific synapse or neuron is stochastically varied to determine its contribution to the output. Here we use a connectomic approach to study the biological implementation of stochastic gradient descent, which requires as-yet unknown circuit structures to inject variability, correlate variability with reward signals, and correctly assign credit to relevant synapses.Node perturbation is conceptually similar to behavioral trial-and-error reinforcement learning (RL) 16,17 . In the vertebrate brain RL is thought to involve the basal ganglia 18 , where a multitude of sensory and other context and state signals converge with action and outcome signals to determine which actions in which state lead to the best outcomes. In the songbird, the basal ganglia circuit dedicated to song learning 19 , Area X, receives synaptic input from two cortical areas 20 : the voc...

Connectomes across development reveal principles of brain maturation

“…Moreover, multiple animals need to be analyzed to assess structural and behavioural heterogeneity. Serial-section electron microscopy (EM) allows reconstruction of neural circuits with synapse resolution [14][15][16][17][18][19][20] , but low throughput makes it difficult to compare whole brain samples and comprehensively assess plasticity. EM has been applied to assess wiring differences between species 21 , sexes 22 , genotypes 23 , and ages 4,24 .…”

Multiscale and multimodal reconstruction of cortical structure and function

“…Connectomic reconstructions of neural tissue using electron microscopy (EM) are becoming more broadly available, and have been used to study ultrastructural features of specific neuronal cell types (Ribak and Anderson, 1980; Spacek and Harris, 1997; Karube, Kubota and Kawaguchi, 2004; Wu et al ., 2017), connectivity rules (Kasthuri et al ., 2015; Lee et al ., 2016), plasticity rules (Bartol et al ., 2015; Bloss et al ., 2018; Dorkenwald et al ., 2019; Kornfeld et al ., 2020), circuit development (Wilson et al ., 2019), and sensory processing (Kim et al ., 2014; Takemura, Nern, et al ., 2017; Vishwanathan et al ., 2017; Wanner and Friedrich, 2020). Reconstructions have probed circuitry in various parts of the nervous system for multiple species, including songbird (Kornfeld et al ., 2017, 2020), zebrafish (Vishwanathan et al ., 2017; Wanner and Friedrich, 2020), rodent retina, cortex, and cerebellum (Helmstaedter et al ., 2013; Kasthuri et al ., 2015; Behrens et al ., 2016; Morgan, 2017; Schmidt et al ., 2017; Bae et al ., 2018; Dorkenwald et al ., 2019; Motta et al ., 2019; Wilson et al ., 2019; Schneider- Mizell et al ., 2020), as well as Drosophila melanogaster (Takemura, Aso, et al ., 2017; Zheng et al ., 2018; Dorkenwald et al ., 2020; Shan Xu et al ., 2020).…”

Neuronal Subcompartment Classification and Merge Error Correction

“…Base FFN supervoxels (SVs) were agglomerated ( Fig. 1a) via FFN resegmentation, with additional post-processing applied to the agglomeration graph to reduce merge and split errors [9]. We also used precomputed organelle probability maps for synaptic junctions and vesicle clouds [4] in some experiments.…”

“…Semi-automated reconstructions of these volumes yield thousands of neurons and neuronal fragments, interconnected by millions of synapses [4][5][6][7]. Together, reconstructed neurons and synapses within each dataset describe a "connectome": a connectivity graph whose structure is anticipated to underlie the computational function of the tissue [8,9].…”