Reward-Dependent Modulation of Movement Variability

Pekny, Sarah E.; Izawa, Jun; Shadmehr, Reza

doi:10.1523/jneurosci.3244-14.2015

Cited by 156 publications

(250 citation statements)

References 57 publications

Supporting

Mentioning

218

Contrasting

Order By: Relevance

“…Before considering how this hypothesis can be incorporated in a computational model of choice behavior, we examined a more direct reflection of the participants' sensitivity to the movement feedback. Motor execution errors should drive adaptation of the movements themselves, resulting in subtle trial-by-trial changes in reach direction subsequent to miss trials in the Spatial condition (10). To confirm the presence of adaptation, we analyzed how the error on trial t affected the reach direction on the next trial t + 1.…”

Section: Resultsmentioning

confidence: 99%

Credit assignment in movement-dependent reinforcement learning

McDougle

Boggess

Crossley

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model wherein movement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tractable solution to the credit assignment problem.decision-making | reinforcement learning | sensory prediction error | reward prediction error | cerebellum W hen a diner reaches across the table and knocks over her coffee, the absence of anticipated reward should be attributed to a failure of coordination rather than diminish her love of coffee. Although this attribution is intuitive, current models of decision-making lack a mechanistic explanation for this seemingly simple computation. We set out to ask if, and how, selection processes in decision-making incorporate information specific to action execution and thus solve the credit assignment problem that arises when an expected reward is not obtained because of a failure in motor execution.Humans are highly capable of tracking the value of stimuli, varying their behavior on the basis of reinforcement history (1, 2), and exhibiting sensitivity to intrinsic motor noise when reward outcomes depend on movement accuracy (3-5). In real-world behavior, the underlying cause of unrewarded events is often ambiguous: A lost point in tennis could occur because the player made a poor choice about where to hit the ball or failed to properly execute the stroke. However, in laboratory studies of reinforcement learning, the underlying cause of unrewarded events is typically unambiguous, either solely dependent on properties of the stimulus or on motor noise. Thus, it remains unclear how people assign credit to either extrins...

show abstract

Section: Resultsmentioning

confidence: 99%

Credit assignment in movement-dependent reinforcement learning

McDougle

Boggess

Crossley

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Dopamine depletion made PD patients less willing to invest effort compared to controls (Chong et al 2015), implicitly affecting estimation of movement energy cost and therefore movement speed (Mazzoni et al 2007; Moisello et al 2011b). Movement variability is reduced in PD, probably because of impaired sensitivity to negative outcomes (with preserved sensitivity to positive outcomes), determining less exploratory variation in motor performance (Pekny et al 2015). …”

Section: Reward and Learningmentioning

confidence: 99%

The many facets of motor learning and their relevance for Parkinson's disease

Marinelli

Quartarone

Hallett

et al. 2017

Clinical Neurophysiology

109

111

View full text Add to dashboard Cite

The final goal of motor learning, a complex process that includes both implicit and explicit (or declarative) components, is the optimization and automatization of motor skills. Motor learning involves different neural networks and neurotransmitters systems depending on the type of task and on the stage of learning. After the first phase of acquisition, a motor skill goes through consolidation (i.e., becoming resistant to interference) and retention, processes in which sleep and long-term potentiation seem to play important roles. The studies of motor learning in Parkinson’s disease have yielded controversial results that likely stem from the use of different experimental paradigms. When a task’s characteristics, instructions, context, learning phase and type of measures are taken into consideration, it is apparent that, in general, only learning that relies on attentional resources and cognitive strategies is affected by PD, in agreement with the finding of a fronto-striatal deficit in this disease. Levodopa administration does not seem to reverse the learning deficits in PD, while deep brain stimulation of either globus pallidus or subthalamic nucleus appears to be beneficial. Finally and most importantly, patients with PD often show a decrease in retention of newly learned skill, a problem that is present even in the early stages of the disease. A thorough dissection and understanding of the processes involved in motor learning is warranted to provide solid bases for effective medical, surgical and rehabilitative approaches in PD.

show abstract

“…We can modulate the degree of variability used during motor learning, as either part of a reward [45] or error-based learning process [46], and this modulation in variability eventually decreases once we master a new motor skill [47][48][49]. Movement variability is also critical for responding to permutations in the environment and producing on-line corrections [44,50,51].…”

Section: Introductionmentioning

confidence: 99%

Methods for Measuring Swallowing Pressure Variability Using High-Resolution Manometry

Jones

Meisner

Broadfoot

et al. 2018

Front. Appl. Math. Stat.

View full text Add to dashboard Cite

Any movement performed repeatedly will be executed with inter-trial variability. Oropharyngeal swallowing is a complex sensorimotor action, and swallow-to-swallow variability can have consequences that impact swallowing safety. Our aim was to determine an appropriate method to measure swallowing pressure waveform variability. An ideal variability metric must be sensitive to known deviations in waveform amplitude, duration, and overall shape, without being biased by waveforms that have both positive and sub-atmospheric pressure profiles. Through systematic analysis of model waveforms, we found a coefficient of variability (CV) parameter on waveforms adjusted such that the overall mean was 0 to be best suited for swallowing pressure variability analysis. We then investigated pharyngeal swallowing pressure variability using high-resolution manometry data from healthy individuals to assess impacts of waveform alignment, pharyngeal region, and number of swallows investigated. The alignment that resulted in the lowest overall swallowing pressure variability was when the superior-most sensor in the upper esophageal sphincter (UES) reached half its maximum pressure. Pressures in the tongue base region of the pharynx were least variable and pressures in the hypopharynx region were most variable. Sets of 3-10 consecutive swallows had no overall difference in variability, but sets of two swallows resulted in significantly less variability than the other dataset sizes. This study identified variability in swallowing pressure waveform shape throughout the pharynx in healthy adults; we discuss implications for swallowing motor control.

show abstract

Reward-Dependent Modulation of Movement Variability

Cited by 156 publications

References 57 publications

Credit assignment in movement-dependent reinforcement learning

Credit assignment in movement-dependent reinforcement learning

The many facets of motor learning and their relevance for Parkinson's disease

Methods for Measuring Swallowing Pressure Variability Using High-Resolution Manometry

Contact Info

Product

Resources

About