TEACh: Task-driven Embodied Agents that Chat

Abstract

Robots operating in human spaces must be able to engage in natural language interaction, both understanding and executing instructions, and using conversation to resolve ambiguity and correct mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human-human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates in natural language with a Follower. The Follower navigates through and interacts with the environment to complete tasks varying in complexity from MAKE COFFEE to PREPARE BREAKFAST, asking questions and getting additional information from the Commander. We propose three benchmarks using TEACh to study embodied intelligence challenges, and we evaluate initial models' abilities in dialogue understanding, language grounding, and task execution.

Evaluating Multimodal Interactive Agents

“…Other work has been done on grounded language agents but their capabilities are restricted to a limited task space (Bara et al, 2021;Thomason et al, 2019;Padmakumar et al, 2022). However, recent work in simulation and robotics using human demonstration data has pushed the capabilities of grounded language agents toward open-ended, language-conditioned interaction (Lynch et al, 2020;Abramson et al, 2020;DeepMind Interactive Agents Team et al, 2021;Padmakumar et al, 2021). To our knowledge, this work presents the first careful study of a metric that evaluates open-ended interaction in a standardised way.…”

DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following

“…ALFRED [2], a recently proposed benchmark along this direction, requires the agent to complete complex household tasks by following natural language instructions. Dialogue-enabled agents in navigation or manipulation tasks have recently been proposed [14], [15] -these focus on action prediction from dialogue history, and do not emphasize the agent's ability to ask taskappropriate questions. In this paper, we take a further step in dialogue-enabled agents by presenting a benchmark for the agent to actively ask questions and learn from the answers to better finish the task.…”

Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale

“…Embodied Demonstrations from Humans. Prior expert demonstration datasets for embodied tasks combining vision and action (and optionally language) can be broadly categorized into either consisting of shortest-path trajectories from a planner with privileged information [5,7,8,29], or consisting of human-provided trajectories [23][24][25]. While some works in the former collect natural language data from hu-mans [5,7], we contend that collecting navigation data from humans is equally crucial.…”

“…Datasets with human-provided navigation trajectories are typically small. TEACh [23], CVDN [24] and WAY [25] have <10k episodes, while the EmbodiedQA [8] dataset has ∼700 human-provided episodes -all prohibitively small for training proficient agents. A key contribution of our work is a scalable webbased infrastructure for collecting human navigation and interaction demonstrations, that is easily extensible to any task situated in the Habitat [19] simulator, including languagebased tasks.…”

“…In total we collect 92k human demonstrations, 80k demonstrations for OBJECTNAV and 12k demonstrations for PICK&PLACE. In contrast, the largest existing datasets have 3-10k human demonstrations in simulation [23][24][25] or on real robots [26,27], an order of magnitude smaller. This virtual teleoperation data contains 29.3M actions, which is equivalent to 22, 600 hours of real-world teleoperation time assuming a LoCoBot motion model from [28] (details in appendix (Sec.…”