Learning the microstructure of successful behavior

Abstract

Reinforcement signals indicating success or failure are known to alter the probability of selecting between distinct actions. However, successful performance of many motor skills, such as speech articulation, also requires learning behavioral trajectories that vary continuously over time. Here, we investigated how temporally discrete reinforcement signals shape a continuous behavioral trajectory, the fundamental frequency of adult Bengalese finch song. We provided reinforcement contingent on fundamental frequency performance only at one point in song. Learned changes to fundamental frequency were maximal at this point, but also extended both earlier and later in the fundamental frequency trajectory. A simple principle predicted the detailed structure of learning; birds learn to produce the average of the behavioral trajectories associated with successful outcomes. This learning rule accurately predicts the structure of learning at a millisecond time scale, demonstrating that the nervous system records fine-grained details of successful behavior and uses this information to guide learning.

Vocal learning is constrained by the statistics of sensorimotor experience

“…4A, Right) results in less reinforcement. This account of our results is consistent with recent studies that use white noisebased negative reinforcement signals to drive adaptive changes in pitch (10,24). These studies show that when a subset of pitches is punished by an aversive white noise stimulus, songbirds change their pitch distribution to avoid white noise.…”

“…In white noise experiments, a successful outcome would be an escape from negative reinforcement. Indeed, experimental conditions in which no trials are successful (very large pitch shifts in our study or previously published experiments in which white noise is provided on every trial) produce zero or near-zero learning (10,24). Therefore, despite the methodological differences between shifting the pitch of feedback and differentially reinforcing some syllable variants with white noise, we believe that learning in both paradigms can be explained by birds evaluating the relative value of a range of vocal pitches.…”

Variability in action: Contributions of a songbird cortical-basal ganglia circuit to vocal motor learning and control

“…In adults, recent work indicates that the song variability continues to be significant for song plasticity. Trial-by-trial variability in adult song can enable reinforcement learning, and the degree of variability can constrain the amount of plasticity (Tumer and Brainard, 2007;Andalman and Fee, 2009;Charlesworth et al, 2011;Warren et al, 2011;Sober and Brainard, 2012).…”

Rhythmic Continuous-Time Coding in the Songbird Analog of Vocal Motor Cortex

“…Continuous activity in HVC (RA) neurons also figures into recent models of sequence generation in which these neurons form a synaptically connected chain that supports the propagation of bursting activity (Glaze and Troyer, 2007;Hanuschkin et al, 2011;Jin et al, 2007;Li and Greenside, 2006;Long and Fee, 2008;Long et al, 2010). Finally, a recent model of vocal learning incorporates X-projecting HVC (HVC (X) ) neurons to form a continuous-time code (or clock), allowing temporal specificity in learning (Charlesworth et al, 2011;Fee and Goldberg, 2011;Ravbar et al, 2012).…”