Actor-critic reinforcement learning in the songbird

Chen, Ruidong; Goldberg, Jesse H.

doi:10.1016/j.conb.2020.08.005

Cited by 33 publications

(31 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, in adult birds song learning is first implemented with neuroplastic changes in premotor circuits, but over the course of days is consolidated with neuroplastic changes in RA, the song-specialized pallial M1-analog (134,135). The authors propose that this two-step consolidation process mitigates "catastrophic forgetting", the tendency of reinforcement learning algorithms to forget old information as new information is learned (133). Whether or not proposed reinforcement learning neural circuits in other species also contain two-step consolidation processes is an open question worth exploring.…”

Section: Reward Prediction Error and Motor Skill Learningmentioning

confidence: 99%

New roles for dopamine in motor skill acquisition: lessons from primates, rodents, and songbirds

Wood

2021

Journal of Neurophysiology

View full text Add to dashboard Cite

Motor learning is a core aspect of human life, and appears to be ubiquitous throughout the animal kingdom. Dopamine, a neuromodulator with a multifaceted role in synaptic plasticity, may be a key signaling molecule for motor skill learning. Though typically studied in the context of reward-based associative learning, dopamine appears to be necessary for some types of motor learning. Mesencephalic dopamine structures are highly conserved among vertebrates, as are some of their primary targets within the basal ganglia, a subcortical circuit important for motor learning and motor control. With a focus on the benefits of cross-species comparisons, this review examines how "model-free" and "model-based" computational frameworks for understanding dopamine's role in associative learning may be applied to motor learning. The hypotheses that dopamine could drive motor learning either by functioning as a reward prediction error, through passive facilitating of normal basal ganglia activity, or through other mechanisms are examined in light of new studies using humans, rodents, and songbirds. Additionally, new paradigms that could enhance our understanding of dopamine's role in motor learning by bridging the gap between the theoretical literature on motor learning in humans and other species are discussed.

show abstract

Section: Reward Prediction Error and Motor Skill Learningmentioning

confidence: 99%

New roles for dopamine in motor skill acquisition: lessons from primates, rodents, and songbirds

Wood

2021

Journal of Neurophysiology

View full text Add to dashboard Cite

show abstract

“…These, moreover, are depleted by experienced pain (and replenished by outcome-driven positive affect). It should be possible to isolate and study such representations, using the same techniques that have yielded evidence of prediction error and confidence being represented in the basal ganglia dopamine system (e.g., Gershman and Uchida 2019 ; Chen and Goldberg 2020 ). Although the representations of bidding strength must be distinct from those of dopamine-mediated reinforcement learning credit, they may still involve dopamine, as suggested by the recent finding that in the dorsal tegmentum, dopamine neurons gate associative learning of fear ( Groessl et al 2018 ).…”

Section: An Evolutionary-functional Take On Painmentioning

confidence: 99%

A possible evolutionary function of phenomenal conscious experience of pain

Kolodny

Moyal

Edelman

2021

Neuroscience of Consciousness

View full text Add to dashboard Cite

Evolutionary accounts of feelings, and in particular of negative affect and of pain, assume that creatures that feel and care about the outcomes of their behavior outperform those that do not in terms of their evolutionary fitness. Such accounts, however, can only work if feelings can be shown to contribute to fitness-influencing outcomes. Simply assuming that a learner that feels and cares about outcomes is more strongly motivated than one that does is not enough, if only because motivation can be tied directly to outcomes by incorporating an appropriate reward function, without leaving any apparent role to feelings (as it is done in state-of-the-art engineered systems based on reinforcement learning). Here, we propose a possible mechanism whereby pain contributes to fitness: an actor-critic functional architecture for reinforcement learning, in which pain reflects the costs imposed on actors in their bidding for control, so as to promote honest signaling and ultimately help the system optimize learning and future behavior.

show abstract

“…Nonetheless, reinforcement-based models for song learning largely focus on learning features of song syllables rather than syllable sequences (Fee and Goldberg, 2011 ; Fee, 2012 ; Chen and Goldberg, 2020 ; Kornfeld et al, 2020 ). Therefore, from a circuit perspective, it is not clear how chunked motor programs associated with syllable level representations might be acquired during development.…”

Section: Introductionmentioning

confidence: 99%

“…Although the neuronal basis for how Area X MSNs integrate their various inputs remains largely unknown, local credit assignment models provide at least one basis for thinking about how reinforcement learning shapes synaptic plasticity and guides song learning in Area X (Fee and Goldberg, 2011 ; Fee, 2012 ; Chen and Goldberg, 2020 ; Kornfeld et al, 2020 ). These models posit that coincident signals from HVC, LMAN, and VTA/SN, in a manner following three-factor Hebbian learning rules (Kuśmierz et al, 2017 ), drive plastic changes at corticostriatal synapses.…”

Section: Introductionmentioning

confidence: 99%

What Is the Role of Thalamostriatal Circuits in Learning Vocal Sequences?

Xiao

Roberts

2021

Front. Neural Circuits

View full text Add to dashboard Cite

Basal ganglia (BG) circuits integrate sensory and motor-related information from the cortex, thalamus, and midbrain to guide learning and production of motor sequences. Birdsong, like speech, is comprised of precisely sequenced vocal elements. Learning song sequences during development relies on Area X, a vocalization related region in the medial striatum of the songbird BG. Area X receives inputs from cortical-like pallial song circuits and midbrain dopaminergic circuits and sends projections to the thalamus. It has recently been shown that thalamic circuits also send substantial projections back to Area X. Here, we outline a gated-reinforcement learning model for how Area X may use signals conveyed by thalamostriatal inputs to direct song learning. Integrating conceptual advances from recent mammalian and songbird literature, we hypothesize that thalamostriatal pathways convey signals linked to song syllable onsets and offsets and influence striatal circuit plasticity via regulation of cholinergic interneurons (ChIs). We suggest that syllable sequence associated vocal-motor information from the thalamus drive precisely timed pauses in ChIs activity in Area X. When integrated with concurrent corticostriatal and dopaminergic input, this circuit helps regulate plasticity on medium spiny neurons (MSNs) and the learning of syllable sequences. We discuss new approaches that can be applied to test core ideas of this model and how associated insights may provide a framework for understanding the function of BG circuits in learning motor sequences.

show abstract

Actor-critic reinforcement learning in the songbird

Cited by 33 publications

References 42 publications

New roles for dopamine in motor skill acquisition: lessons from primates, rodents, and songbirds

New roles for dopamine in motor skill acquisition: lessons from primates, rodents, and songbirds

A possible evolutionary function of phenomenal conscious experience of pain

What Is the Role of Thalamostriatal Circuits in Learning Vocal Sequences?

Contact Info

Product

Resources

About