The MAGICAL Benchmark for Robust Imitation

Toyer, Sam; Shah, Rohin; Critch, Andrew; Russell, Stuart

doi:10.48550/arxiv.2011.00401

Cited by 3 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Environments and training data. We evaluate on ten tasks taken from three benchmark domains: DMC [8], Procgen [7], and MAGICAL [6]. Here we briefly explain our choice of tasks and datasets; for more detailed information (e.g.…”

Section: Experiments Setupmentioning

confidence: 99%

“…For each MAGICAL environment, we used a fixed subset of five demonstration trajectories (initially selected at random) from the human dataset provided with the benchmark [6]. We used egocentric views with a frame stack of four and no action repeat.…”

Section: Hyperparametermentioning

confidence: 99%

“…As previous work has found that image augmentation alone can outperform more complex representation learning techniques [4,5], we make sure to compare against baselines that use augmentation. To ensure generalizability of our results, we evaluate on ten tasks selected across three benchmarks, including MAGICAL [6], Procgen [7] and the DeepMind Control Suite (DMC) [8].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Empirical Investigation of Representation Learning for Imitation

Chen¹,

Toyer²,

Wild³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Imitation learning often needs a large demonstration set in order to handle the full range of situations that an agent might find itself in during deployment. However, collecting expert demonstrations can be expensive. Recent work in vision, reinforcement learning, and NLP has shown that auxiliary representation learning objectives can reduce the need for large amounts of expensive, task-specific data. Our Empirical Investigation of Representation Learning for Imitation (EIRLI) investigates whether similar benefits apply to imitation learning. We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation across several environment suites. In the settings we evaluate, we find that existing algorithms for image-based representation learning provide limited value relative to a well-tuned baseline with image augmentations. To explain this result, we investigate differences between imitation learning and other settings where representation learning has provided significant benefit, such as image classification. Finally, we release a well-documented codebase which both replicates our findings and provides a modular framework for creating new representation learning algorithms out of reusable components.

show abstract

Section: Experiments Setupmentioning

confidence: 99%

Section: Hyperparametermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Investigation of Representation Learning for Imitation

Chen¹,

Toyer²,

Wild³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We introduce a cross-embodiment imitation learning benchmark, X-MAGICAL, which is based on the imitation learning benchmark MAGICAL [33], implemented on top of the physics engine PyMunk [34].…”

Section: X-magical Benchmarkmentioning

confidence: 99%

XIRL: Cross-embodiment Inverse Reinforcement Learning

Zakka¹,

Zeng²,

Florence³

et al. 2021

Preprint

View full text Add to dashboard Cite

We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings has typically required alignment with a reference trajectory, which may be difficult to acquire. We show empirically that if the embeddings are aware of task-progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. We also find that XIRL policies are more sample efficient than baselines, and in some cases exceed the sample efficiency of the same agent trained with ground truth sparse rewards.

show abstract

“…Recently, with the growing interest of the research community for IL, LfD or ORL, a lot of datasets have been released. They mainly focus on robotics [15,44,48,40], some with a particular focus on human data [31,32,39]. Some works include datasets for discrete action-environments like games [19,28].…”

Section: Related Workmentioning

confidence: 99%

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

Ramos¹,

Girgin²,

Hussenot³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce RLDS (Reinforcement Learning Datasets), an ecosystem for recording, replaying, manipulating, annotating and sharing data in the context of Sequential Decision Making (SDM) including Reinforcement Learning (RL), Learning from Demonstrations, Offline RL or Imitation Learning. RLDS enables not only reproducibility of existing research and easy generation of new datasets, but also accelerates novel research. By providing a standard and lossless format of datasets it enables to quickly test new algorithms on a wider range of tasks. The RLDS ecosystem makes it easy to share datasets without any loss of information and to be agnostic to the underlying original format when applying various data processing pipelines to large collections of datasets. Besides, RLDS provides tools for collecting data generated by either synthetic agents or humans, as well as for inspecting and manipulating the collected data. Ultimately, integration with TFDS [3] facilitates the sharing of RL datasets with the research community.

show abstract

The MAGICAL Benchmark for Robust Imitation

Cited by 3 publications

References 10 publications

An Empirical Investigation of Representation Learning for Imitation

An Empirical Investigation of Representation Learning for Imitation

XIRL: Cross-embodiment Inverse Reinforcement Learning

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

Contact Info

Product

Resources

About