"Good Robot!": Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer

Abstract

In order to learn effectively, robots must be able to extract the intangible context by which task progress and mistakes are defined. In the domain of reinforcement learning, much of this information is provided by the reward function. Hence, reward shaping is a necessary part of how we can achieve state-of-the-art results on complex, multi-step tasks. However, comparatively little work has examined how reward shaping should be done so that it captures task context, particularly in scenarios where the task is long-horizon and failure is highly consequential. Our Schedule for Positive Task (SPOT) reward trains our Efficient Visual Task (EVT) model to solve problems that require an understanding of both task context and workspace constraints of multi-step block arrangement tasks. In simulation EVT can completely clear adversarial arrangements of objects by pushing and grasping in 99% of cases vs an 82% baseline in prior work. For random arrangements EVT clears 100% of test cases at 86% action efficiency vs 61% efficiency in prior work. EVT + SPOT is also able to demonstrate context understanding and complete stacks in 74% of trials compared to a baseline of 5% with EVT alone. To our knowledge, this is the first instance of a Reinforcement Learning based algorithm successfully completing such a challenge. Code is available at https://github.com/jhu-lcsr/good robot.

Evaluation of Techniques for Sim2Real Reinforcement Learning

“…There are two classes of DA: instance-based and feature-based. DA is utilized in (Bousmalis et al 2018;Yan et al 2017;Hundt et al 2020) research to optimize robot grasping. Research has shown that the DA method allows the model to learn a mapping from source to target domain (Tobin et al 2017).…”

Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives

“…The benefit of those models is their ability to project a goal image and their current observation into their feature space and compute a path towards the target feature for visual servo-ing (Watter et al, 2015;Byravan et al, 2018), reaching and pushing (Srinivas et al, 2018;Yu et al, 2019) with gradientbased optimisation methods. Visuomotor controllers trained in the reinforcement learning paradigm typically model the distance to a desired, visually specified goal via reward functions which can be either shaped explicitly based on expert domain knowledge (Hundt et al, 2019) or implicitly learned from user feedback about task success (Singh et al, 2019). Our approach of using dynamic images for goal distance estimation sets itself apart from these methods as it uses dynamic images as an efficient, non-parametric conditioning scheme.…”

Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey

“…PyBullet and MuJoCo, on the other hand, present wider integration with DL and RL libraries and gym environments. In In those cases where system identification for one-shot transfer is the objective, researchers have often built or customized specific simulations that meet problem-specific requirements and constraints [32], [36], [41].…”