Learning to Walk Via Deep Reinforcement Learning

Abstract

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website 3 .

ToriLLE: Learning Environment for Hand-to-Hand Combat

“…While SAC also uses weight of the entropy loss to encourage exploration, the implementation also includes automatic entropy scaling [24]. However, we found this automatic tuning to very quickly set the entropy weight very low, and agent was not able to improve after that point.…”

DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning

“…[9] demonstrated that blind locomotion controllers could be transferred to real systems by incorporating actuator dynamics into offline training in simulation. Moreover, [10] showed that training was also possible directly on hardware. However, the aforementioned systems, crafted and trained end-to-end are limited to operating blind and on flat-terrain.…”

Sim-to-Real Transfer for Biped Locomotion

“…Despite concerns about safety and sample complexity of DRL methods, there has been success in directly training locomotion controllers on the real robot [32], [33]. In Ha et al [33], a policy was directly trained on a multi-legged robot.…”

“…The training was automated by a novel resetting device which was able to re-initialize the robot during training after each rollout. In Haarnoja et al [32], a policy was trained for a real quadruped robot in under two hours from scratch using softactor critic algorithm [34]. Despite these success in learning legged locomotion tasks, directly training policies on a biped robot is still challenging due to the frequent manual resetting required during training and the potential safety concern from the inherent instability.…”