Exploration-based learning of a step to step controller predicts locomotor adaptation

Abstract

Humans are able to adapt their locomotion to a variety of novel circumstances, for instance, walking on diverse terrain and walking with new footwear. During locomotor adaptation, humans have been shown to exhibit stereotypical changes in their movement patterns. Here, we provide a theoretical account of such locomotor adaptation, positing that the nervous system prioritizes stability in the short timescale and improves energy expenditure over a longer timescale. The resulting mathematical model has two processes: a stabilizing controller which is gradually changed by a reinforcement learner that exploits local gradients to lower energy expenditure, estimating gradients indirectly via intentional exploratory noise. We consider this model walking and adapting under three novel circumstances: walking on a split-belt treadmill (walking with each foot on a different belt, each belt at different speeds), walking with an exoskeleton, and walking with an asymmetric leg mass. This model predicts the short and long timescale changes observed in walking symmetry on the split-belt treadmill and while walking with the asymmetric mass. The model exhibits energy reductions with exoskeletal assistance, as well as entrainment to time-periodic assistance. We show that such exploration-based learning is degraded in the presence of large sensorimotor noise, providing a potential account for some impairments in learning.

The split-belt rimless wheel

“…This evolution of step length asymmetry is associated with a decrease in energy cost between early and late adaptation, consistent with the theory that people are adapting to take advantage of positive work performed by the treadmill. This trend toward positive asymmetry during split-belt adaptation has also been observed in previous work modeling split-belt walking with bipedal robots (Fujiki et al, 2015; Otoda et al, 2009; Simha, 2020; Seethapathi et al, 2021).…”

“…Our work adds to the body of literature using simulation models to understand human split-belt walking. Previous work has shown that bipedal robots on split-belt treadmills adapt from negative asymmetry toward positive asymmetry (Fujiki et al, 2015; Otoda et al, 2009; Simha, 2020; Seethapathi et al, 2021); our paper helps explain why this adaptation could be beneficial in allowing the treadmill to do work on the person. More complicated simulation models, as well as future human-subject experiments, can further reveal the opportunities and limitations for humans to take advantage of the split-belt treadmill.…”

“…The frameworks from these original models enabled later experimental investigations of the relative contributions of step-to-step transition losses (Donelan et al, 2002) and swing costs (Doke et al, 2005), and analysis of how ankle and hip power contribute to propulsion (Kuo, 2002; Lewis and Ferris, 2008). Bipedal models have previously been used to explore split-belt walking, from modeling the role of the cerebellum in split-belt adaptation (Fujiki et al, 2015; Otoda et al, 2009) to attempts to define an optimization function that predicts human walking behavior in different circumstances (Seethapathi et al, 2021; Simha, 2020). Simulation models allow exploration of an entire parameter space in a way not possible in human experiments because of time constraints.…”

Forward simulations of walking on a variable surface-impedance treadmill: A comparison of two methods

“…While these two modeling approaches are useful, there still exists a need for a robust model which can simulate longterm adaptations to persistent foot-ground contact asymmetry. Methods such as single-shooting optimal control [26], [27] and reinforcement learning [28], [29] are better suited for simulating interactive perturbations and observing the change in behavior and may be appropriate for modeling the immediate aftermath of large stiffness or damping perturbations. There is also a need for new experiments using a VST to perform prolonged perturbations with additional EMG and kinematic data for both lower limbs, and to perform long term adaptation studies in general.…”

Habitual exercise evokes fast and persistent adaptation during split-belt walking

“…Adaptation of SLA during split-belt walking occurs at two distinct rates -a fast component that adapts rapidly and a slow component that adapts more gradually (Darmohray et al, 2019;Mawase et al, 2013;Roemmich et al, 2016;Sánchez et al, 2021). Recently, Seethapathi and colleagues computationally posited that the fast component that adapts rapidly is driven by balance optimization, whereas the slow component that adapts more gradually is driven by energetic cost optimization (Seethapathi et al, 2021). SLA adaptation during split-belt walking occurs in parallel with reductions in the work generated by the legs; gait adaptation results in decreasing positive work and increasing negative work done by the legs, especially the leg on the fast belt (Sánchez et al, 2019;Selgrade et al, 2017).…”