SAPIEN: A SimulAted Part-Based Interactive ENvironment

Xiang, Fanbo; Qin, Yuzhe; Mo, Kaichun; Xia, Yikuan; Zhu, Hao; Liu, Fangchen; Liu, Minghua; Jiang, Hanxiao; Yuan, Yifu; Wang, He; Yi, Li; Chang, Angel X.; Guibas, Leonidas J.; Su, Hao

doi:10.1109/cvpr42600.2020.01111

Cited by 275 publications

(213 citation statements)

References 24 publications

Supporting

Mentioning

205

Contrasting

Order By: Relevance

“…Terry Winograd -1971 high fidelity simulators for robotics (Todorov et al, 2012;Coumans andBai, 2016-2019;NVIDIA, 2019;Xiang et al, 2020) and the cost and availability of commodity hardware (Fitzgerald, 2013;Campeau-Lecours et al, 2019;Murali et al, 2019).…”

Section: Ws4: Embodiment and Actionmentioning

confidence: 99%

Experience Grounds Language

Bisk¹,

Holtzman²,

Thomason³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

161

136

View full text Add to dashboard Cite

Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates. Despite the incredible effectiveness of language processing models to tackle tasks after being trained on text alone, successful linguistic communication relies on a shared experience of the world. It is this shared experience that makes utterances meaningful.Natural language processing is a diverse field, and progress throughout its development has come from new representational theories, modeling techniques, data collection paradigms, and tasks. We posit that the present success of representation learning approaches trained on large, text-only corpora requires the parallel tradition of research on the broader physical and social context of language to address the deeper questions of communication.

show abstract

Section: Ws4: Embodiment and Actionmentioning

confidence: 99%

Experience Grounds Language

Bisk¹,

Holtzman²,

Thomason³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

161

136

View full text Add to dashboard Cite

show abstract

“…Compared to robot simulation with abstract action (Kolve et al 2017;Savva et al 2019;, full-physic robot simulation supports low-level policy learning that could be transferred to real world. Therefore, we integrate our depth simulator with state-ofthe-art robot simulator SAPIEN (Xiang et al 2020), targeting at vision-based robot policy learning. To the best of our knowledge, our work is the first physics-grounded depth simulator integrated in robot simulation environment.…”

Section: Robot Simulationmentioning

confidence: 99%

“…Specifically, we simulate the full processing flow from infrared (IR) pattern projection and rendering, to IR noise simulation and depth reconstruction. We build our pipeline based on SAPIEN (Xiang et al 2020) as it already has state-of-the-art physical simulators and user-friendly robotic interfaces integrated so that we can focus on the sensor simulation.…”

Section: Active Stereovision Depth Sensor Simulationmentioning

confidence: 99%

“…One of the depth sensors is fixed on the table (referred as base RealSense) and another one is mounted to the robot end effector(referred as hand RealSense). We further build the simulation environment in SAPIEN (Xiang et al 2020) and align the reality and simulation as described in Section 5.1. Figure 8 shows the alignment result.…”

Section: Experiments Setupmentioning

confidence: 99%

“…The depth sensor simulator is able to generate low visual domain gap depth maps for objects of different materials. It has been integrated with the SAPIEN robot physic simulation platform (Xiang et al 2020) and will be open-sourced for community usage.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Zhang¹,

Chen²,

Xiang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by the underlying mechanism of the sensors, we designed a fully physics-grounded simulation pipeline, which includes material acquisition, ray tracing based infrared (IR) image rendering, IR noise simulation, and depth estimation. The pipeline is able to generate depth maps with material-dependent error patterns similar to a real depth sensor. We conduct extensive experiments to show that perception algorithms and reinforcement learning policies trained in our simulation platform could transfer well to real world test cases without any fine-tuning. Furthermore, due to the high degree of realism of this simulation, our depth sensor simulator can be used as a convenient testbed to evaluate the algorithm performance in the real world, which will largely reduce the human effort in developing robotic algorithms. The entire pipeline has been integrated into the SAPIEN simulator and is open-sourced to promote the research of vision and robotics communities.

show abstract