Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing

Abstract

Robot simulation has been an essential tool for data-driven manipulation tasks. However, most existing simulation frameworks lack either efficient and accurate models of physical interactions with tactile sensors or realistic tactile simulation. This makes the sim-to-real transfer for tactile-based manipulation tasks still challenging. In this work, we integrate simulation of robot dynamics and vision-based tactile sensors by modeling the physics of contact. This contact model uses simulated contact forces at the robot's end-effector to inform the generation of realistic tactile outputs. To eliminate the simto-real transfer gap, we calibrate our physics simulator of robot dynamics, contact model, and tactile optical simulator with real-world data, and then we demonstrate the effectiveness of our system on a zero-shot sim-to-real grasp stability prediction task where we achieve an average accuracy of 90.7% on various objects. Experiments reveal the potential of applying our simulation framework to more complicated manipulation tasks. We open-source our simulation framework at https://github.com/CMURoboTouch/Taxim/tree/taxim-robot.

Toward Zero-Shot Sim-to-Real Transfer Learning for Pneumatic Soft Robot 3D Proprioceptive Sensing

“…Robotics researchers in fields such as manipulation and aerial locomotion have utilized simulated environments to train deep learning visual models [13,14,24]. In such cases, sim-to-real transfer process involves reducing the difference between the simulation-based rendered images and the realworld images through careful simulation scene adjustments based on some reference real-world images [13,24] or by filtering the images to reduce the impact of hard-to-model optical effects in the renderer [14].…”

“…In other robotics applications that similarly suffer from difficulty in collecting reliable training data in the real world, researchers have successfully taken advantage of simulations to collect the data and transferred the knowledge to the real world in the form of trained deep learning models [13,14]. Such sim-to-real transfer learning approaches not only allow data collection with ease but also enable observations under conditions that are physically impossible to observe in the real world.…”