Hierarchical Reinforcement Learning for Quadruped Locomotion

Abstract

Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hierarchical framework to automatically decompose complex locomotion tasks. A high-level policy issues commands in a latent space and also selects for how long the low-level policy will execute the latent command. Concurrently, the low-level policy uses the latent command and only the robot's on-board sensors to control the robot's actuators. Our approach allows the high-level policy to run at a lower frequency than the lowlevel one. We test our framework on a path-following task for a dynamic quadruped robot and we show that steering behaviors automatically emerge in the latent command space as low-level skills are needed for this task. We then show efficient adaptation of the trained policy to a different task by transfer of the trained low-level policy. Finally, we validate the policies on a real quadruped robot. To the best of our knowledge, this is the first application of end-to-end hierarchical learning to a real robotic locomotion task.

Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

“…While encouraging results have been achieved using Model Predictive Control (MPC) and trajectory optimization [24,10,18,9,19,26,4,75], these methods require in-depth knowledge of the environment and substantial efforts in manual parameter tuning, which makes these methods challenging to apply to complex environments. Alternatively, model-free RL can learn general policies for tasks with challenging terrain [43,90,53,63,64,77,35,46,85,36,38,84,44]. For example, Xie et al [85] introduce to use dynamics randomization to generalize RL locomotion policy in different environments, and Peng et al [64] use animal videos to provide demonstrations for imitation learning.…”

“…Finally, we combine both features for policy action prediction. The resulting model is trained end-to-end directly using rewards, without hierarchical RL [62,41,31,38] nor pre-defined controllers [15,21].…”

Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space

“…Hierarchical Reinforcement Learning (HRL). [9], [10] is widely used for robotic tasks, such as manipulation [5], navigation [6], and locomotion [26]. In this work, we refrain from learning the low-level policy parameters, hence our method is not HRL-based.…”

Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning

“…Hierarchical Reinforcement Learning. Hierarchical Reinforcement Learning (HRL) [25], [26] has also proven to be an effective tool in tackling the locomotion problem. Furthermore, researchers [27] have developed HRL strategies for controlling legged characters in simulation, which have been further combined with adversarial learning [28] to achieve high-level control.…”