Understanding Domain Randomization for Sim-to-real Transfer

Abstract

Reinforcement learning encounters many challenges when applied directly in the real world. Sim-to-real transfer is widely used to transfer the knowledge learned from simulation to the real world. Domain randomization-one of the most popular algorithms for sim-to-real transfer-has been demonstrated to be effective in various tasks in robotics and autonomous driving. Despite its empirical successes, theoretical understanding on why this simple algorithm works is limited. In this paper, we propose a theoretical framework for sim-to-real transfers, in which the simulator is modeled as a set of MDPs with tunable parameters (corresponding to unknown physical parameters such as friction). We provide sharp bounds on the sim-to-real gap-the difference between the value of policy returned by domain randomization and the value of an optimal policy for the real world. We prove that sim-to-real transfer can succeed under mild conditions without any real-world training samples. Our theory also highlights the importance of using memory (i.e., history-dependent policies) in domain randomization. Our proof is based on novel techniques that reduce the problem of bounding the sim-to-real gap to the problem of designing efficient learning algorithms for infinite-horizon MDPs, which we believe are of independent interest.

Multi-View Metal Parts Pose Estimation Based on a Single Camera

“…The images are generated by running the physics simulation Blender 2.79 (Blender Foundation, Amsterdam, Netherlands). for a few seconds using domain randomization [ 37 ], generating physical object configurations. The images are photorealistic.…”

Linguistic Explanations of Black Box Deep Learning Detectors on Simulated Aerial Drone Imagery

“…Thankfully, due to tools like the Unreal Engine, Unity [ 25 ], Apple’s Hypersim [ 26 ], and NVIDIA’s Isaac Gym [ 27 ], we can now quickly produce large and diverse sets of photorealistic imagery with dense and accurate truth. While these tools are not a perfect simulation or a replacement for the real-world, their level of realism is high and multiple works are already showing how they can be used as is to train and evaluate AI/ML algorithms [ 28 , 29 , 30 , 31 , 32 , 33 ]. In [ 34 ], we showed how this imagery can be used to systematically evaluate model performance.…”

A Review of Deep Reinforcement Learning Approaches for Smart Manufacturing in Industry 4.0 and 5.0 Framework

“…Domain randomisation. To accommodate a wider variety of environmental setups and potential scenarios, simulation parameters are randomly generated [226]. This includes two groups of techniques that bridge the gap between the actual world and the virtual one.…”