MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Fan, Linxi; Wang, Guanzhi; Jiang, Yunfan; Mandlekar, Ajay; Yue, Yang; Zhu, Hua; Tang, Andrew; Huang, De-An; Zhu, Yuke; Anandkumar, Anima

doi:10.48550/arxiv.2206.08853

Cited by 14 publications

(32 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We adopt the original observation space provided by MineDoJo [17], which includes a RGB camera-view, yaw/pitch angle, GPS location, and the type of 3 × 3 blocks surrounding the agent. We discretize the original multidiscrete action space provided by MineDojo into 42 discrete actions.…”

Section: Methodsmentioning

confidence: 99%

“…First, MineAgent [17] is an online RL algorithm that leverages pretrained state representations and dense reward functions to boost training. BC (VPT) [4], BC (CLIP) [17], and BC (I-CNN) [15] are variants of the behavior cloning algorithm that use different backbone models (indicated in the corresponding brackets) for state feature extraction. The backbones are finetuned with the BC loss (see Appendix A.3 for more details).…”

Section: Single Task Experimentsmentioning

confidence: 99%

“…Building agents that can accomplish a vast and diverse suite of tasks in an open-ended world is considered a key challenge towards devising generally capable artificial intelligence [2,3,6,32]. In recent years, environments like Minecraft have drawn much attention from the related research communities [17,[19][20][21]26], since they are not only [50] (left) and Minecraft [24] (right) based on t-SNE visualization. The points with the same color represent states from the trajectories that complete the same task.…”

Section: Introductionmentioning

confidence: 99%

“…Another grand challenge in an open-ended environment like Minecraft hails from the setting of such games, where an agent can only have very limited observations of the world. For example, in MineDoJo [17] (a recent agent benchmark built on Minecraft), the observation space comprises a first-person view image and a list of possessed items. However, many more aspects of the surroundings remain hidden from the agents.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction

Cai¹,

Wang²,

Ma³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study the problem of learning goal-conditioned policies in Minecraft, a popular, widely accessible yet challenging open-ended environment for developing human-level multi-task agents. We first identify two main challenges of learning such policies: 1) the indistinguishability of tasks from the state distribution, due to the vast scene diversity, and 2) the non-stationary nature of environment dynamics caused by partial observability. To tackle the first challenge, we propose Goal-Sensitive Backbone (GSB) for the policy to encourage the emergence of goal-relevant visual state representations. To tackle the second challenge, the policy is further fueled by an adaptive horizon prediction module that helps alleviate the learning uncertainty brought by the non-stationary dynamics. Experiments on 20 Minecraft tasks show that our method significantly outperforms the best baseline so far; in many of them, we double the performance. Our ablation and exploratory studies then explain how our approach beat the counterparts and also unveil the surprising bonus of zero-shot generalization to new scenes (biomes). We hope our agent could help shed some light on learning goal-conditioned, multi-task agents in challenging, open-ended environments like Minecraft. The code is released at https://github.com/CraftJarvis/ MC-Controller.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Single Task Experimentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction

Cai¹,

Wang²,

Ma³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…On the other hand, our simulator is more suitable for open-world exploration. Recently, MINEDOJO has been developed with thousands of diverse open-ended tasks [32]. With MINEDOJO's data, one can leverage large pre-trained video language models to learn reward functions and then guide agent learning in various tasks.…”

Section: Related Workmentioning

confidence: 99%

WILD-SCAV: Benchmarking FPS Gaming AI on Unity3D-based Environments

Chen¹,

Shi²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances in deep reinforcement learning (RL) have demonstrated complex decision-making capabilities in simulation environments such as Arcade Learning Environment [1], MuJoCo [2] and ViZDoom [3]. However, they are hardly extensible to more complicated problems, mainly due to the lack of complexity and variations in the environments they are trained and tested on. Furthermore, they are not extensible to an open world environment to facilitate long-term exploration research. To learn realistic task-solving capabilities, we need to develop an environment with greater diversity and complexity. We developed WILD-SCAV, a powerful and extensible environment based on a 3D open-world FPS (First-Person Shooter) game to bridge the gap. It provides realistic 3D environments of variable complexity, various tasks, and multiple modes of interaction, where agents can learn to perceive 3D environments, navigate and plan, compete and cooperate in a human-like manner. WILD-SCAV also supports different complexities, such as configurable maps with different terrains, building structures and distributions, and multi-agent settings with cooperative and competitive tasks. The experimental results on configurable complexity, multi-tasking, and multi-agent scenarios demonstrate the effectiveness of WILD-SCAV in benchmarking various RL algorithms, as well as it is potential to give rise to intelligent agents with generalized task-solving abilities. The link to our open-sourced code can be found here 1 .

show abstract

KSRL: Knowledge Selection Based Reinforcement Learning for Knowledge-Grounded Dialogue

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Cited by 14 publications

References 0 publications

Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction

Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction

WILD-SCAV: Benchmarking FPS Gaming AI on Unity3D-based Environments

KSRL: Knowledge Selection Based Reinforcement Learning for Knowledge-Grounded Dialogue

Contact Info

Product

Resources

About