Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Abstract

Reinforcement learning provides a general framework for learning robotic skills while minimizing engineering effort. However, most reinforcement learning algorithms assume that a well-designed reward function is provided, and learn a single behavior for that single reward function. Such reward functions can be difficult to design in practice. Can we instead develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then re-purpose these skills for downstream tasks? In this paper, we demonstrate that a recently proposed unsupervised skill discovery algorithm can be extended into an efficient off-policy method, making it suitable for performing unsupervised reinforcement learning in the real world. Firstly, we show that our proposed algorithm provides substantial improvement in learning efficiency, making rewardfree real-world training feasible. Secondly, we move beyond the simulation environments and evaluate the algorithm on real physical hardware. On quadrupeds, we observe that locomotion skills with diverse gaits and different orientations emerge without any rewards or demonstrations. We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.

Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning

“…The concept of mutual information, which is also at the heart of empowerment based methods, has been further used to motivate several objectives for skill discovery (Florensa et al, 2017;Eysenbach et al, 2019;Achiam et al, 2018;Warde-Farley et al, 2019;Hansen et al, 2020;Sharma et al, 2020b). Recent works have shown that skills learned through mutual information can be meaningfully combined to solve downstream tasks (Eysenbach et al, 2019;Sharma et al, 2020b), even on real robots (Sharma et al, 2020a).…”

Artificial intelligence-controlled pole balancing using an Arduino board

“…It is also claimed that RL is an effective training method for these systems. The work of (Sharma et al, 2020) uses a free-reward RL algorithm to teach a robot how to travel properly within its structure and navigate, 20 hours of preparation, the machine had mastered a variety of locomotion gaits. After a short time, the RL methods yielded results.…”

Reset-Free Lifelong Learning with Skill-Space Planning

“…Intuitively, the skill-practice distribution can alleviate this, associating states with certain skills which are important, making the skill learning process easier. We investigate this by analyzing DADS-Off (Sharma et al, 2020a) -an off-policy improved version of DADS -and resampling skills every K timesteps. This simulates K-step rollouts from arbitrary starting states, i.e.…”

“…In particular, we can consider the intrinsic reward, which is a proxy for the diversity of skills. Since the intrinsic reward is calculated under the model, which changes and has inaccuracies, high intrinsic reward under the model is not always the best indicator, whereas it is a more reliable metric when learned from real world transitions as in Sharma et al (2020a). Also, the intrinsic reward is sampled according to an expectation given by the skill-practice distribution, so these numbers cannot be directly compared with those given by the Sharma et al (2020a) paper.…”