Why Build an Assistant in Minecraft?

Szlam, Arthur; Gray, Jonathan; Srinet, Kavya; Jernite, Yacine; Joulin, Armand; Synnaeve, Gabriel; Kiela, Douwe; Yu, Haonan; Chen, Zhuoyuan; Goyal, Siddharth; Guo, Demi; Rothermel, Danielle; Zitnick, C. Lawrence; Weston, Jason

doi:10.48550/arxiv.1907.09273

Cited by 4 publications

(4 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Work in this area encompasses image captioning (Lin et al, 2014), video captioning (Yu et al, 2016), visual QA (Antol et al, 2015), and more conversationally, visual dialogue (Das et al, 2017). Embodied agents that use language are also being explored (Das et al, 2018;Savva et al, 2019;Szlam et al, 2019;Urbanek et al, 2019).…”

Section: Multimodality and Groundingmentioning

confidence: 99%

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions

Roller,

Boureau,

Weston

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We present our view of what is necessary to build an engaging open-domain conversational agent: covering the qualities of such an agent, the pieces of the puzzle that have been built so far, and the gaping holes we have not filled yet. We present a biased view, focusing on work done by our own group, while citing related work in each area. In particular, we discuss in detail the properties of continual learning, providing engaging content, and being well-behaved -and how to measure success in providing them. We end with a discussion of our experience and learnings, and our recommendations to the community.

show abstract

Section: Multimodality and Groundingmentioning

confidence: 99%

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions

Roller,

Boureau,

Weston

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In contrast to other experimentation systems, such as Johnson et al (2016) (who modify the Minecraft client) or Szlam et al (2019) (who use the thirdparty server Cuberite), we make use of the official Minecraft server, which means that users can use an unmodified up-to-date Minecraft client. Experiments can also make use of all features introduced by new Minecraft releases, if they wish.…”

Section: The Minecraft Servermentioning

confidence: 99%

“…This feature-richness makes Minecraft a perfect environment for the evaluation of all kinds of intelligent agents (Johnson et al, 2016), from reinforcement learning agents (Guss et al, 2019), to instruction receiving (Szlam et al, 2019) and instruction giving assistants (Narayan-Chen et al, 2019). Its popularity (Minecraft is the most sold game of all time), together with the client-server architecture make Minecraft a tool well-suited for crowd-sourcing with volunteers from all over the world.…”

Section: Introductionmentioning

confidence: 99%

MC-Saar-Instruct: a Platform for Minecraft Instruction Giving Agents

Köhn,

Wichlacz,

Schäfer

et al. 2020

Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

We present a comprehensive platform to run human-computer experiments where an agent instructs a human in Minecraft, a 3D blocksworld environment. This platform enables comparisons between different agents by matching users to agents. It performs extensive logging and takes care of all boilerplate, allowing to easily incorporate new agents to evaluate them. Our environment is prepared to evaluate any kind of instruction giving system, recording the interaction and all actions of the user. We provide example architects, a Wizardof-Oz architect and set-up scripts to automatically download, build and start the platform.

show abstract

“…A number of machine learning (ML) research projects about following instructions tasks have been initiated by making use of the video game Minecraft (Johnson et al, 2016;Shu et al, 2018;Narayan-Chen et al, 2019;Guss et al, 2019;Jayannavar et al, 2020). Building such agents requires to make progress in grounded natural language understanding -understanding complex instructions, for example, with spatial relations in natural languageself-improvement -studying how to flexibly learn from human interactions -synergies of ML components -exploring the integration of several ML and non-ML components to make them work together (Szlam et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

Learning to Execute Actions or Ask Clarification Questions

Shi¹,

Feng²,

Lipani³

2022

Preprint

View full text Add to dashboard Cite

Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly.

show abstract

Why Build an Assistant in Minecraft?

Abstract: In this document we describe a rationale for a research program aimed at building an open "assistant" in the game Minecraft, in order to make progress on the problems of natural language understanding and learning from dialogue.

Cited by 4 publications

References 65 publications

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions

MC-Saar-Instruct: a Platform for Minecraft Instruction Giving Agents

Learning to Execute Actions or Ask Clarification Questions

Contact Info

Product

Resources

About