Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Chen, Rui; Li, Ang; Xiang, Fanbo; Qin, Yuzhe; Gu, Jiwei; Ling, Zhan; Liu, Minghua; Zeng, Peiyu; Han, Songfang; Huang, Zhiao; Mu, Tongzhou; Xu, Jing; Su, Hao

doi:10.1109/tro.2023.3235591

Cited by 13 publications

(3 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A major obstacle is the high demand for massive amounts of interaction data, which are costly and time-consuming to collect in reality. Sim2Real addresses this problem by enabling robots to be trained in simulation and transferring the learned policies to real robots with minimal or zero real-world data [7], [8], [9]. Furthermore, the access to ground-truth environment states in simulation accelerates the policy learning process.…”

Section: Zero-shot Sim2realmentioning

confidence: 99%

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Chen,

Xu,

Xiang

et al. 2024

IEEE Trans. Robot.

Self Cite

View full text Add to dashboard Cite

Visuotactile sensors can provide rich contact information, having great potential in contact-rich manipulation tasks with reinforcement learning (RL) policies. Sim2Real technique tackles the challenge of RL's reliance on a large amount of interaction data. However, most Sim2Real methods for manipulation tasks with visuotactile sensors rely on rigidbody physics simulation, which fails to simulate the real elastic deformation precisely. Moreover, these methods do not exploit the characteristic of tactile signals for designing the network architecture. In this paper, we build a general-purpose Sim2Real protocol for manipulation policy learning with marker-based visuotactile sensors. To improve the simulation fidelity, we employ an FEM-based physics simulator that can simulate the sensor deformation accurately and stably for arbitrary geometries. We further propose a novel tactile feature extraction network that directly processes the set of pixel coordinates of tactile sensor markers and a self-supervised pre-training strategy to improve the efficiency and generalizability of RL policies. We conduct extensive Sim2Real experiments on the peg-in-hole task to validate the effectiveness of our method. And we further show its generalizability on additional tasks including plug adjustment and lock opening. The protocol, including the simulator and the policy learning framework, will be open-sourced for community usage.

show abstract

Section: Zero-shot Sim2realmentioning

confidence: 99%

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Chen,

Xu,

Xiang

et al. 2024

IEEE Trans. Robot.

Self Cite

View full text Add to dashboard Cite

show abstract

“…RFUniverse supports a set of physics realistic sensors to equip the agent with the capability of perception. For visual input, we leverage ray-tracing techniques and integrated the IR-based depth rendering proposed in Sapien [50] to RFUniverse, which mimics the sensor noise of IR-based depth sensors like RealSense D415 camera. Besides, we also notice the trending of vision-based tactile sensing research in the robotics community [34,8,42,39,37].…”

Section: Multi-modal Sensingmentioning

confidence: 99%

Demonstrating RFUniverse: A Multiphysics Simulation Platform for Embodied AI

Fu,

Xu,

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

Multiphysics phenomena, the coupling effects involving different aspects of physics laws, are pervasive in the real world and can often be encountered when performing everyday household tasks. Intelligent agents which seek to assist or replace human laborers will need to learn to cope with such phenomena in household task settings. To equip the agents with such kind of abilities, the research community needs a simulation environment, which will have the capability to serve as the testbed for the training process of these intelligent agents, to have the ability to support multiphysics coupling effects.Though many mature simulation software for multiphysics simulation have been adopted in industrial production, such techniques have not been applied to robot learning or embodied AI research. To bridge the gap, we propose a novel simulation environment named RFUniverse. This simulator can not only compute rigid and multi-body dynamics, but also multiphysics coupling effects commonly observed in daily life, such as air-solid interaction, fluid-solid interaction, and heat transfer.Because of the unique multiphysics capacities of this simulator, we can benchmark tasks that involve complex dynamics due to multiphysics coupling effects in a simulation environment before deploying to the real world. RFUniverse provides multiple interfaces to let the users interact with the virtual world in various ways, which is helpful and essential for learning, planning, and control. We benchmark three tasks with reinforcement learning, including food cutting, water pushing, and towel catching. We also evaluate butter pushing with a classic planningcontrol paradigm. This simulator offers an enhancement of physics simulation in terms of the computation of multiphysics coupling effects. The simulation environment, videos, and other supplementary materials can be viewed on the website: https: //sites.google.com/view/rfuniverse.

show abstract

“…Image Labels: To generate synthetic image observations (RGB and depth) as well as pose and instance segmentation labels from the precomputed GIGA scenes, we use the raytracingbased renderer from SAPIEN [13] and its realistic depth feature [37]. All the textures, materials, lights, and table shapes are randomized, see Fig.…”

Section: Data Generationmentioning

confidence: 99%

Correct Me If I am Wrong: Interactive Learning for Robotic Manipulation

Chisari

Welschehold

Boedecker

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Reliable object grasping is a crucial capability for autonomous robots. However, many existing grasping approaches focus on general clutter removal without explicitly modeling objects and thus only relying on the visible local geometry. We introduce CenterGrasp, a novel framework that combines object awareness and holistic grasping. CenterGrasp learns a general object prior by encoding shapes and valid grasps in a continuous latent space. It consists of an RGB-D image encoder that leverages recent advances to detect objects and infer their pose and latent code, and a decoder to predict shape and grasps for each object in the scene. We perform extensive experiments on simulated as well as real-world cluttered scenes and demonstrate strong scene reconstruction and 6-DoF grasppose estimation performance. Compared to the state of the art, CenterGrasp achieves an improvement of 38.5 mm in shape reconstruction and 33 percentage points on average in grasp success. We make the code and trained models publicly available at http://centergrasp.cs.uni-freiburg.de.

show abstract

Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Cited by 13 publications

References 74 publications

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

General-Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors

Demonstrating RFUniverse: A Multiphysics Simulation Platform for Embodied AI

Correct Me If I am Wrong: Interactive Learning for Robotic Manipulation

Contact Info

Product

Resources

About