Building a state space for song learning

Abstract

The songbird system has shed light on how the brain produces precisely timed behavioral sequences, and how the brain implements reinforcement learning (RL). RL is a powerful strategy for learning what action to produce in each state, but requires a unique representation of the states involved in the task. Songbird RL circuitry is thought to operate using a representation of each moment within song syllables, consistent with the sparse sequential bursting of neurons in premotor cortical nucleus HVC. However, such sparse sequences are not present in very young birds, which sing highly variable syllables of random lengths. Here, we review and expand upon a model for how the songbird brain could construct latent sequences to support RL, in light of new data elucidating connections between HVC and auditory cortical areas. We hypothesize that learning occurs via four distinct plasticity processes: 1) formation of 'tutor memory' sequences in auditory areas; 2) formation of appropriately-timed latent HVC sequences, seeded by inputs from auditory areas spontaneously replaying the tutor song; 3) strengthening, during spontaneous replay, of connections from HVC to auditory neurons of corresponding timing in the 'tutor memory' sequence, aligning auditory and motor representations for subsequent song evaluation; and 4) strengthening of connections from premotor neurons to motor output neurons that produce the desired sounds, via well-described song RL circuitry.

Threshold-Based Ordering of Sequential Actions during Drosophila Courtship

“…A ramp-to-threshold mechanism, coded by spike count, provides a parsimonious explanation of the ordering of the engagement actions. This model offers a flexible and neurally efficient means of generating action sequences, as the actions can be sequentially evoked without any neural mechanism to ensure their mutual exclusivity or the feedforward signaling that is invoked in other models [1,[3][4][5][6]9]. If multiple neurons each contribute spikes to be counted by downstream circuits, then eliminating just one class of input neurons would be predicted to reduce the frequency of these actions, but not change their order, as we observed upon aSP22 ablation.…”

“…Thus, as courtship progresses into engagement, actions begin sequentially and persist cumulatively; they are not mutually exclusive. Escalating, overlapping actions are not explicitly featured in most models of sequential behavior [1][2][3][4][5][6][7][8][9][10]. Additional models based on various forms of inhibition, in mammalian striatum and fly larvae [4,8,10], also do not explicitly model overlapping actions, as seen here.…”

“…Work in multiple species has led to several theories of how nervous systems organize sequences of motor actions to form complex behaviors [1][2][3][4][5][6][7][8][9][10]. Nevertheless, evidence to support these theories has often been correlational; in few systems have specific cell types been discovered to be both necessary and sufficient for the execution of a behavioral sequence, partially due to the historical difficulty of controlling individual cell types.…”

Songbird ventral basal ganglia sends performance error signals to dopaminergic midbrain

“…Foremost, it remains unclear how AIV computes its auditory error signal. Extracting this error signal may require comparison with tutor song, which appears to be stored in other auditory cortical areas (Mackevicius and Fee, 2018b). It may also require comparison of actual acoustic signals against predicted acoustic signals, which could rely on a forward model of song production (e.g.…”

“…Yet in other respects, parts of the song system appear to embody the AC architecture (Doya and Sejnowski, 1995). The premotor cortical nucleus HVC can generate state-like information in the form of what time-step it is in the song (Hahnloser et al, 2002;Mackevicius and Fee, 2018a). VTA DA neurons resemble the critic's output; they signal performance prediction error, the difference between how good a syllable just sounded and how good it was predicted to sound based on recent performance ( Figure 1C) (Gadagkar et al, 2016).…”