Dynamics Randomization Revisited: A Case Study for Quadrupedal Locomotion

Abstract

Understanding the gap between simulation and reality is critical for reinforcement learning with legged robots, which are largely trained in simulation. However, recent work has resulted in sometimes conflicting conclusions with regard to which factors are important for success, including the role of dynamics randomization. In this paper, we aim to provide clarity and understanding on the role of dynamics randomization in learning robust locomotion policies for the Laikago quadruped robot. Surprisingly, in contrast to prior work with the same robot model, we find that direct simto-real transfer is possible without dynamics randomization or on-robot adaptation schemes. We conduct extensive ablation studies in a sim-to-sim setting to understand the key issues underlying successful policy transfer, including other design decisions that can impact policy robustness. We further ground our conclusions via sim-to-real experiments with various gaits, speeds, and stepping frequencies. Additional Details: pair.toronto.edu/understanding-dr/ *Work done during an internship at NVIDIA 1 NVIDIA,

Robust Value Iteration for Continuous Control Tasks

“…Therefore, the resulting policies are not robust to changes in the dynamics and fail to solve the task in the real world. In contrast, sim2real methods extending RL with domain randomization [1][2][3][4] or adversarial disturbances [5][6][7][8] have shown the successful transfer to the physical world [9].…”

“…In robotics, domain randomization is most widely used to achieve successful sim2real transfer. For example domain randomization was used for in-hand manipulation [46], ball-in-a-cup [2], locomotion [9], manipulation [3,4].…”

Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization

“…However, these methods require in-depth knowledge about the environment and substantial manual efforts for parameter tuning. As an alternative, reinforcement learning provides an autonomous learning paradigm for legged locomotion skills from self-exploration in complex environments [19], [21], [27], [31], [38], [45], [47], [55], [62]. Despite the successful application of RL on legged robots, most RL approaches depend only on proprioceptive input.…”

“…However, only given proprioceptive information, the blind RL controller addresses challenging scenarios by training with large-scale randomized environment parameters [31], [62]. While this technique delivers promising results for maneuvering on uneven and unknown-material ground, it's insufficient for more complicated tasks like avoiding obstacles that are hard to step over, or estimating accurate foot placement positions for safety.…”

Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

“…It expands the reach of robots and enables them to solve a wide range of tasks, from daily life delivery to planetary exploration in challenging, uneven terrain [16,2]. Recently, besides the success of Deep Reinforcement Learning (RL) in navigation [56,27,86,42] and robotic manipulation [49,48,78,40], we have also witnessed the tremendous improvement of locomotion skills for quadruped robots, allowing them to walk on uneven terrain [85,84], and even generalize to real-world with mud, snow, and running water [46]. While these results are encouraging, most RL approaches focus on learning a robust controller for blind quadrupedal locomotion, using only the proprioceptive measurements as inputs.…”