Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system

Abstract

Reinforcement learning has emerged as a promising methodology for training robot controllers. However, most results have been limited to simulation due to the need for a large number of samples and the lack of automated-yet-safe data collection methods. Model-based reinforcement learning methods provide an avenue to circumvent these challenges, but the traditional concern has been the mismatch between the simulator and the real world. Here, we show that control policies learned in simulation can successfully transfer to a physical system, composed of three Phantom robots pushing an object to various desired target positions. We use a modified form of the natural policy gradient algorithm for learning, applied to a carefully identified simulation model. The resulting policies, trained entirely in simulation, work well on the physical system without additional training. In addition, we show that training with an ensemble of models makes the learned policies more robust to modeling errors, thus compensating for difficulties in system identification. The results are illustrated in the accompanying video.

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

“…Combination of system identification and dynamics randomization has been used in the past to learn locomotion for a real quadruped [26], non-prehensile object manipulation [27] and in-hand object pivoting [28]. In our work, we recognize domain randomization and system identification as powerful tools for training general policies in simulation.…”

Grounded action transformation for sim-to-real reinforcement learning

“…Manual parameter tuning is another form of simulator modification that can be done prior to applying reinforcement learning. Lowrey et al (2018) manually identify simulation parameters before applying policy gradient reinforcement learning to learn to push an object to target positions. Tan et al (2018) perform similar system identification (including disassembling the robot and making measurements of each part) and adding action latency modeling before using deep reinforcement learning to learn quadrapedal walking.…”

“…Domain randomization produces policies that are robust enough to transfer to the real world. An alternative approach that does not involve randomness is to learn policies that perform well under an ensemble of different simulators (Boeing & Bräunl, 2012;Rajeswaran et al, 2017;Lowrey et al, 2018). Pinto et al, (2017b) simultaneously learn an adversary that can perturb the learning agent's actions while it learns in simulation.…”

Benchmarking In-Hand Manipulation

“…for v ∈ V do 10 P r .append(min-vertex(v,O)) 11 for v ∈ P r do 12 min d← ∞ 13 for p ∈ V c do 14 d← G(v, p) 15 if d<min d then 16 min d←d 17 if min d>max d then 18 max d←min d 19 return max d with the object, and the mesh of the desired contact region C d . To compute these metrics, we first project the desired contact region C d and the robot meshes L i∈N onto the object mesh O as shown in lines 5-10.…”