iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes

Shen, Bokui; Xia, Fei; Li, Chengshu; Martí­n-Martí­n, Roberto; Fan, Linxi; Wang, Guanzhi; Pérez-D’Arpino, Claudia; Buch, Shyamal; Srivastava, Sanvesh; Tchapmi, Lyne P.; Tchapmi, Micael; Vainio, Kent; Wong, Josiah; Li, Feifei; Savarese, Silvio

doi:10.48550/arxiv.2012.02924

Cited by 26 publications

(29 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4), we create five realistic multi-stage simulated household kitchen tasks and collect a large-scale multi-user demonstration dataset. We design the simulated tasks in a realistic kitchen environment using PyBullet [60] and the iGibson [61,62] framework with a Fetch [63] robot that must manipulate a bowl. Across all tasks, the robot's initial pose and bowl location is randomized between episodes.…”

Section: Simulated Kitchen Datasetmentioning

confidence: 99%

“…Both the teleoperator and agent are limited to on-board sensors, and are not provided any privileged information such as low-level object states or a global camera view: the imitation learning policy uses as input only RGB-D images and laser scan point clouds. The visual observations provided by the simulator, iGibson [62] are high-quality, with natural textures and materials rendered with a physics-based renderer, background and lighting probes obtained from real world. Moreover, the model used for our simulated robot closely aligns with the real Fetch robot, with identical sensor specifications, kinematics, and control schemes.…”

Section: A10 Sim2real Potentialmentioning

confidence: 99%

See 1 more Smart Citation

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Wong¹,

Tung²,

Kurenkov³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In mobile manipulation (MM), robots can both navigate within and interact with their environment and are thus able to complete many more tasks than robots only capable of navigation or manipulation. In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks. Much prior work has shown that IL can train visuo-motor policies for either manipulation or navigation domains, but few works have applied IL to the MM domain. Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed. To address these problems, we first propose MOBILE MANIPULATION ROBOTURK (MOMART), a novel teleoperation framework allowing simultaneous navigation and manipulation of mobile manipulators, and collect a first-of-its-kind large scale dataset in a realistic simulated kitchen setting. We then propose a learned error detection system to address covariate shift by detecting when an agent is in a potential failure state. We train performant IL policies and error detectors from this data, and achieve over 45% task success rate and 85% error detection success rate across multiple multi-stage tasks when trained on expert data. Codebase, datasets, visualization, and more available at https://sites.google.com/view/il-for-mm/home.

show abstract

Section: Simulated Kitchen Datasetmentioning

confidence: 99%

Section: A10 Sim2real Potentialmentioning

confidence: 99%

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Wong¹,

Tung²,

Kurenkov³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hypersim [16] and OpenRooms [17] develop simulators for indoor object detection. Robotic simulators include AI-2THOR [18], Habitat [19,20], NVIDIA Isaac Sim [21], and iGibson [22] focus largely on embodied AI tasks. More generic tools for object detection dataset generation include BlenderProc [23], BlendTorch [24], NVISII [25], and the Unity Perception package [7].…”

Section: Related Workmentioning

confidence: 99%

PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision

Ebadi¹,

Jhang²,

Zook³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, person detection and human pose estimation have made great strides, helped by large-scale labeled datasets. However, these datasets had no guarantees or analysis of human activities, poses, or context diversity. Additionally, privacy, legal, safety, and ethical concerns may limit the ability to collect more human data. An emerging alternative to real-world data that alleviates some of these issues is synthetic data. However, creation of synthetic data generators is incredibly challenging and prevents researchers from exploring their usefulness. Therefore, we release a human-centric synthetic data generator PEOPLESANSPEO-PLE which contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels. Using PEOPLESANSPEOPLE, we performed benchmark synthetic data training using a Detectron2 Keypoint R-CNN variant [1]. We found that pre-training a network using synthetic data and fine-tuning on target real-world data (few-shot transfer to limited subsets of COCO-person train [2]) resulted in a keypoint AP of 60.37 ± 0.48 (COCO test-dev2017) outperforming models trained with the same real data alone (keypoint AP of 55.80) and pre-trained with ImageNet (keypoint AP of 57.50). This freely-available data generator 1 should enable a wide range of research into the emerging field of simulation to real transfer learning in the critical area of human-centric computer vision.

show abstract

“…In the past few years, researchers have developed many simulation environments [6,7,13,5,9] to serve as training and evaluation platforms for embodied agents. These simulation environments propel research progress in a wide range of embodied tasks, including vision-and-language task completion [10,30], rearrangement [12,7], navigation [9,13], manipulation [31,32] and human-robot collaboration [5]. Recently, AllenAct [33] integrates a set of embodied environments (such as iThor, RoboThor, Habitat [9], etc.…”

Section: Related Workmentioning

confidence: 99%

“…Embodied artificial intelligence (EAI) has attracted significant attention, both in advanced deep learning models and algorithms [1,2,3,4] and the rapid development of simulated platforms [5,6,7,8,9]. Many open challenges [10,11,12,13] have been proposed to facilitate EAI research. A critical bottleneck in existing simulated platforms [10,12,8,5,14] is the limited number of indoor scenes that support vision-and-language navigation, object interaction, and complex household tasks.…”

Section: Introductionmentioning

confidence: 99%

LUMINOUS: Indoor Scene Generation for Embodied AI Challenges

Zhao¹,

Lin²,

Jia³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning-based methods for training embodied agents typically require a large number of high-quality scenes that contain realistic layouts and support meaningful interactions. However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts. This paper presents LUMINOUS, the first research framework that employs stateof-the-art indoor scene synthesis algorithms to generate large-scale simulated scenes for Embodied AI challenges. Further, we automatically and quantitatively evaluate the quality of generated indoor scenes via their ability to support complex household tasks. LUMINOUS incorporates a novel scene generation algorithm (Constrained Stochastic Scene Generation (CSSG)), which achieves competitive performance with human-designed scenes. Within LUMINOUS, the EAI task executor, task instruction generation module, and video rendering toolkit can collectively generate a massive multimodal dataset of new scenes for the training and evaluation of Embodied AI agents. Extensive experimental results demonstrate the effectiveness of the data generated by LUMINOUS, enabling the comprehensive assessment of embodied agents on generalization and robustness. The full codebase and documentation of LUMINOUS is available at: https: //github.com/amazon-research/indoor-scene-generation-eai/.

show abstract

iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes

Cited by 26 publications

References 63 publications

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision

LUMINOUS: Indoor Scene Generation for Embodied AI Challenges

Contact Info

Product

Resources

About