ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Mu, Tongzhou; Ling, Zhan; Xiang, Fanbo; Yang, Derek; Li, Xuanlin; Tao, Stone; Huang, Zhiao; Jia, Zhiwei; Su, Hao

doi:10.48550/arxiv.2107.14483

Cited by 4 publications

(10 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Going beyond state vectors, visual inputs provides not only the appearance but also the geometric information of the scene and object. A lot of recent efforts have been made on learning decision making directly from visual inputs [19,20,25,36,56,59,61,69]. For example, Yarats et al [69] propose to utilize an image reconstruction objective jointly with the RL objective to learn visual representation and decision making at the same time.…”

Section: Related Workmentioning

confidence: 99%

“…For example, Yarats et al [69] propose to utilize an image reconstruction objective jointly with the RL objective to learn visual representation and decision making at the same time. Besides learning from images, Mu et al [36] propose to train policy directly taking point clouds as inputs for manipulation tasks using a robot arm. With imitation learning using large-scale expert demonstrations, the learned policy can be generalized to unseen objects during test time.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Wu¹,

Wang²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

indicates equal contributions.ric representation learning objective using our demonstrations. By experimenting with relocating diverse objects in simulation, we show that our approach outperforms baselines with a large margin when manipulating novel objects. We also ablate the importance on 3D object representation learning for manipulation. We include videos, code, and additional information on the project websitehttps://kristery.github.io/ILAD/.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Wu¹,

Wang²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…manipulate novel object instances from a known object category in a predefined way. Specifically, we train a robotic agent to execute an object manipulation task in a simulation environment following the design of ManiSkill [35], but with the actuator being a dexterous hand. We validate that existing state-of-the-art RL algorithms could hardly achieve satisfactory results on this challenging task, while significant progress can be made through imitating the rich demonstrations in HOI4D.…”

Section: Categories Of Action Segmentationmentioning

confidence: 99%

“…Inspired by ManiSkill [35], we build an environment based on SAPIEN [49] simulator. Environment design: the environment use the SAPIEN simulator with timestep set to 0.05.…”

Section: D1 Environment Setupmentioning

confidence: 99%

See 1 more Smart Citation

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

Liu¹,

Liu²,

Che³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present HOI4D, a large-scale 4D egocentric dataset with rich annotations, to catalyze the research of categorylevel human-object interaction. HOI4D consists of 2.4M RGB-D egocentric video frames over 4000 sequences collected by 9 participants interacting with 800 different object instances from 16 categories over 610 different indoor rooms. Frame-wise annotations for panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose and hand action have also been provided, together with reconstructed object meshes and scene point clouds. With HOI4D, we establish three benchmarking tasks to promote category-level HOI from 4D visual signals including semantic segmentation of 4D dynamic point cloud sequences, category-level object pose tracking, and egocentric action segmentation with diverse interaction targets. In-depth analysis shows HOI4D poses great challenges to existing methods and produces huge research opportunities.

show abstract

Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations

Hao¹,

Wang²,

Cao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multimodal demonstrations provide robots with an abundance of information to make sense of the world. However, such abundance may not always lead to good performance when it comes to learning sensorimotor control policies from human demonstrations. Extraneous data modalities can lead to state over-specification, where the state contains modalities that are not only useless for decision-making but also can change data distribution across environments. State over-specification leads to issues such as the learned policy not generalizing outside of the training data distribution. In this work, we propose Masked Imitation Learning (MIL) to address state over-specification by selectively using informative modalities. Specifically, we design a masked policy network with a binary mask to block certain modalities. We develop a bi-level optimization algorithm that learns this mask to accurately filter over-specified modalities. We demonstrate empirically that MIL outperforms baseline algorithms in simulated domains including MuJoCo and a robot arm environment using the Robomimic dataset [1], and effectively recovers the environment-invariant modalities on a multimodal dataset collected on a real robot. Our project website presents supplemental details and videos of our results at: https://tinyurl.com/masked-il

show abstract

ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Cited by 4 publications

References 48 publications

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations

Contact Info

Product

Resources

About