Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Abstract

Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer to the real world. We narrow this reality gap by improving the physics simulator and learning robust policies. We improve the simulation using system identification, developing an accurate actuator model and simulating latency. We learn robust controllers by randomizing the physical environments, adding perturbations and designing a compact observation space. We evaluate our system on two agile locomotion gaits: trotting and galloping. After learning in simulation, a quadruped robot can successfully perform both gaits in the real world.

MPC-Net: A First Principles Guided Policy Search

“…Recent successes in Reinforcement Learning (RL) demonstrate sophisticated walking robot control [1]- [5], yet a large number of policy rollouts need to be collected to reach the required performance level. It is, therefore, common practice to use physics simulators during training and subsequently attempt a sim-to-real transfer [1], [4].…”

Learning Fast Adaptation With Meta Strategy Optimization

“…We compare MSO to two baselines: domain randomization (DR) [10] and strategy optimization with projected universal policy (SO-PUP) [11]. We run ARS for 1500 iterations for all methods and we use a two-dimensional latent space for MSO and SO-PUP.…”

“…The first task is to transfer the policy trained in simulation to the real Minitaur robot. Although we use the nonlinear actuator model from Tan et al [10], the reality gap in our case is still large as we use a different version of Minitaur and we do not perform additional system identification.…”

“…We show that MSO is extremely data efficient (≤ 15 rollouts or 75 seconds of data) to adapt the policies to novel situations in the target environment. We compare our method to two baseline methods: domain randomization [10] and strategy optimization with a projected universal policy [11]. Our results show that MSO outperforms both baselines in the simulated and the real environments.…”

An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Research

“…They can also significantly reduce the time and cost required for repair and maintenance. These features become especially important when testing learning algorithms directly on real hardware [33], [34] where it is essential to have a safe platform to explore various control patterns.…”