Assessing Transferability From Simulation to Reality for Reinforcement Learning

Abstract

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the 'Simulation Optimization Bias' (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results in two different environments show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real system without any additional training on the latter.

DROPO: Sim-to-Real Transfer with Offline Domain Randomization

“…Other works approached the problem by keeping a fixed distribution over the physical parameters and, instead, relying on intelligent sampling techniques to improve generalization: [23] guides training on increasingly harder environment variations, while [24] increases the number of sampled simulated environments until satisfactory transfer behavior is reached.…”

Distilled Domain Randomization

“…They found empirically that sampling domain parameters from a uniform distribution together with applying random forces and regularizing the observation space can be sufficient to cross the reality gap. Aside from to the previous methods, Muratore et al [8] introduce an approach to estimate the transferability of a policy learned from randomized physics simulations. Moreover, the authors propose a meta-algorithm which provides a probabilistic guarantee on the performance loss when transferring the policy between two domains form the same distribution.…”

“…Learning from randomized simulations has shown to be a promising approach for learning robot control policies that transfer to the real world. Examples cover manipulation [1], [2], [3], [4], [5], trajectory optimization [6], continuous control [7], [8], [9], vision [10], [11], [12], [13], and locomotion tasks [14], [15], [16]. Independent of the task, all domain randomization methods can be classified based on the fact if they use target domain data to update the distribution over simulators or not.…”

On the Emergence of Whole-Body Strategies From Humanoid Robot Push-Recovery Learning

“…In particular, collecting the amount of example trajectories required by most state-of-the-art modelfree DRL algorithms is unfeasible for current robots [4]. A common solution consists in resorting to synthetic data based on rigid-body dynamics, addressing the mismatch introduced by the sim-to-real gap in a subsequent stage [5], [6]. Nonetheless, learned behaviors often display unnatural characteristics, such as asymmetric gaits, abrupt motions of the body and limbs, or even unrealistic motions exploiting imperfections and glitches in the physical simulator of choice.…”