Reinforcement Learning with Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotion

Shi, Haojie; Zhang, Bo; Zeng, Hongsheng; Wang, Fan; Dong, Yueqiang; Li, Jiangyong; Wang, Kang; Tian, Hanqin; Meng, Max Q.-H.

doi:10.48550/arxiv.2109.06409

Cited by 2 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kasaei et al use DRL to update the parameters of a CPG-ZMP walk engine and output joint target residuals to adapt to commands and different terrains [36]. For quadrupeds, Shi et al learn both the parameters of a foot trajectory generator as well as joint target residuals to locomote in a variety of environments including stairs and slopes [37].…”

Section: A Related Workmentioning

confidence: 99%

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

Bellegarda

Ijspeert

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

In this letter, we present a method for integrating central pattern generators (CPGs), i.e. systems of coupled oscillators, into the deep reinforcement learning (DRL) framework to produce robust and omnidirectional quadruped locomotion. The agent learns to directly modulate the intrinsic oscillator setpoints (amplitude and frequency) and coordinate rhythmic behavior among different oscillators. This approach also allows the use of DRL to explore questions related to neuroscience, namely the role of descending pathways, interoscillator couplings, and sensory feedback in gait generation. We train our policies in simulation and perform a sim-to-real transfer to the Unitree A1 quadruped, where we observe robust behavior to disturbances unseen during training, most notably to a dynamically added 13.75 kg load representing 115% of the nominal quadruped mass. We test several different observation spaces based on proprioceptive sensing and show that our framework is deployable with no domain randomization and very little feedback, where along with the oscillator states, it is possible to provide only contact booleans in the observation space. Video results can be found at https://youtu.be/xqXHLzLsEV4.

show abstract

Section: A Related Workmentioning

confidence: 99%

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

Bellegarda

Ijspeert

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…This controller used a pattern generator modulating action space and it is not clear if the system could learn the precise movements needed for the environments in our work. Similarly, in work concurrent to ours, evolved pattern generators were combined with RL to traverse challenging terrains [14]. In the ALLSTEPS approach [15], RL was used to train policies that were able to walk over stepping stones requiring precise foot placements.…”

Section: Related Workmentioning

confidence: 99%

Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner

Brakel¹,

Bohez²,

Hasenclever³

et al. 2021

Preprint

View full text Add to dashboard Cite

Dynamic quadruped locomotion over challenging terrains with precise foot placements is a hard problem for both optimal control methods and Reinforcement Learning (RL). Non-linear solvers can produce coordinated constraint satisfying motions, but often take too long to converge for online application. RL methods can learn dynamic reactive controllers but require carefully tuned shaping rewards to produce good gaits and can have trouble discovering precise coordinated movements. Imitation learning circumvents this problem and has been used with motion capture data to extract quadruped gaits for flat terrains. However, it would be costly to acquire motion capture data for a very large variety of terrains with height differences. In this work, we combine the advantages of trajectory optimization and learning methods and show that terrain adaptive controllers can be obtained by training policies to imitate trajectories that have been planned over procedural terrains by a non-linear solver. We show that the learned policies transfer to unseen terrains and can be fine-tuned to dynamically traverse challenging terrains that require precise foot placements and are very hard to solve with standard RL.

show abstract

Reinforcement Learning with Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotion

Cited by 2 publications

References 18 publications

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner

Contact Info

Product

Resources

About