Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

Chebotar, Yevgen; Handa, Ankur; Makoviychuk, Viktor; Macklin, Miles; Issac, Jan; Ratliff, Nathan; Fox, Dieter

doi:10.1109/icra.2019.8793789

Cited by 350 publications

(244 citation statements)

References 26 publications

(34 reference statements)

Supporting

Mentioning

230

Contrasting

Unclassified

Order By: Relevance

“…For example, Christiano et al [9] learn inverse dynamics models from data gradually collected from a real robotics system, while transferring trajectory planning policy from a simulator. Chebotar et al [10] and Zhu et al [11] transfer manipulation policies by iteratively collecting data on the real system and updating a distribution of dynamics parameters for the simulator physics engine. Similar principles work for the problem of humanoid balancing [12].…”

Section: Related Workmentioning

confidence: 99%

Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors

Molchanov

Chen

Hönig

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Quadrotor stabilizing controllers often require careful, model-specific tuning for safe operation. We use reinforcement learning to train policies in simulation that transfer remarkably well to multiple different physical quadrotors. Our policies are low-level, i.e., we map the rotorcrafts' state directly to the motor outputs. The trained control policies are very robust to external disturbances and can withstand harsh initial conditions such as throws. We show how different training methodologies (change of the cost function, modeling of noise, use of domain randomization) might affect flight performance. To the best of our knowledge, this is the first work that demonstrates that a simple neural network can learn a robust stabilizing low-level quadrotor controller (without the use of a stabilizing PD controller) that is shown to generalize to multiple quadrotors. The video of our experiments can be found at https://sites.google.com/ view/sim-to-multi-quad.

show abstract

Section: Related Workmentioning

confidence: 99%

Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors

Molchanov

Chen

Hönig

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…Both are embarrassingly parallel. Moreover, the necessity of finding good domain parameter distribution parameters φ could be alleviated by adapting these distributions, as done in [17].…”

Section: Limitations Of the Presented Methodsmentioning

confidence: 99%

Assessing Transferability From Simulation to Reality for Reinforcement Learning

Muratore¹,

Gienger²,

Peters³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the 'Simulation Optimization Bias' (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results in two different environments show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real system without any additional training on the latter.

show abstract

“…Perturbation can also be seen on robot locomotion [22] where planning is done through an ensemble of perturbed models. Lastly, interleaving policy roll outs between simulation and reality has also proven to work well on swing-peg-in-hole and opening a cabinet drawer tasks [11].…”

Section: Related Workmentioning

confidence: 99%

BayesSim: Adaptive Domain Randomization Via Probabilistic Inference for Robotics Simulators

Ramos¹,

Possas²,

Fox³

2019

Robotics: Science and Systems XV

Self Cite

View full text Add to dashboard Cite

We introduce BayesSim 1 , a framework for robotics simulations allowing a full Bayesian treatment for the parameters of the simulator. As simulators become more sophisticated and able to represent the dynamics more accurately, fundamental problems in robotics such as motion planning and perception can be solved in simulation and solutions transferred to the physical robot. However, even the most complex simulator might still not be able to represent reality in all its details either due to inaccurate parametrization or simplistic assumptions in the dynamic models. BayesSim provides a principled framework to reason about the uncertainty of simulation parameters. Given a black box simulator (or generative model) that outputs trajectories of state and action pairs from unknown simulation parameters, followed by trajectories obtained with a physical robot, we develop a likelihood-free inference method that computes the posterior distribution of simulation parameters. This posterior can then be used in problems where Sim2Real is critical, for example in policy search. We compare the performance of BayesSim in obtaining accurate posteriors in a number of classical control and robotics problems. Results show that the posterior computed from BayesSim can be used for domain randomization outperforming alternative methods that randomize based on uniform priors.

show abstract

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

Cited by 350 publications

References 26 publications

Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors

Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors

Assessing Transferability From Simulation to Reality for Reinforcement Learning

BayesSim: Adaptive Domain Randomization Via Probabilistic Inference for Robotics Simulators

Contact Info

Product

Resources

About