DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following

Abstract

Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively. We present DialFRED, a dialogue-enabled embodied instruction following benchmark based on the ALFRED benchmark. DialFRED allows an agent to actively ask questions to the human user; the additional information in the user's response is used by the agent to better complete its task. We release a humanannotated dataset with 53K task-relevant questions and answers and an oracle to answer questions. To tackle DialFRED, we propose a questioner-performer framework wherein the questioner is pre-trained with the human-annotated data and fine-tuned with reinforcement learning. Experimental results show that asking the right questions leads to significantly improved task performance. We make DialFRED publicly available and encourage researchers to propose and evaluate their solutions to building dialog-enabled embodied agents: https://github.com/anonrabit/DialFRED

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

“…Meanwhile, for tasks (Thomason et al, 2019b;Padmakumar et al, 2021) that do not provide an oracle agent to answer question in natural language, researchers also need to build a rule-based (Padmakumar et al, 2021) or neural-based (Roman et al, 2020) oracle. Dial-FRED (Gao et al, 2022) uses a language model as an oracle to answer questions.…”

“…Language-Active Environment Room-to-Room (Anderson et al, 2018b) Matterport3D ✗ Indoor Room-for-Room Matterport3D ✗ Indoor Room-Across-Room (Ku et al, 2020) Matterport3D ✗ Indoor Landmark-RxR (He et al, 2021) Matterport3D ✗ Indoor XL-R2R (Yan et al, 2020) Matterport3D ✗ Indoor VLNCE (Krantz et al, 2020) Habitat ✗ Indoor StreetLearn Google Street View ✗ Outdoor StreetNav (Hermann et al, 2020) Google Street View ✗ Outdoor TOUCHDOWN Google Street View ✗ Outdoor Talk2Nav (Vasudevan et al, 2021) Google Street View ✗ Outdoor LANI (Misra et al, 2018) -✗ Outdoor RoomNav (Wu et al, 2018) House3D ✗ Indoor EmbodiedQA (Das et al, 2018) House3D ✗ Indoor REVERIE Matterport3D ✗ Indoor SOON (Zhu et al, 2021a) Matterport3D ✗ Indoor IQA (Gordon et al, 2018) AI2-THOR ✗ Indoor CHAI (Misra et al, 2018) CHALET ✗ Indoor ALFRED (Shridhar et al, 2020) AI2-THOR ✗ Indoor VNLA Matterport3D ✓ Indoor HANNA (Nguyen and Daumé III, 2019) Matterport3D ✓ Indoor CEREALBAR -✓ Indoor Just Ask (Chi et al, 2020) Matterport3D ✓ Indoor CVDN (Thomason et al, 2019b) Matterport3D ✓ Indoor RobotSlang (Banerjee et al, 2020) -✓ Indoor Talk the Walk (de Vries et al, 2018) -✓ Outdoor MC Collab (Narayan- Minecraft ✓ Outdoor TEACh (Padmakumar et al, 2021) AI2-THOR ✓ Indoor DialFRED (Gao et al, 2022) AI2-THOR ✓ Indoor…”

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

“…The follower converses with the commander and interacts with the environment to complete various house tasks such as making coffee. Dial-FRED (Gao et al, 2022) extends ALFRED (Shridhar et al, 2020) dataset by allowing the agent to actively ask questions.…”

“…Meanwhile, for tasks (Thomason et al, 2019b;Padmakumar et al, 2021) that do not provide an oracle agent to answer question in natural language, researchers also need to build a rule-based (Padmakumar et al, 2021) or neural-based (Roman et al, 2020 oracle. DialFRED (Gao et al, 2022) uses a language model as an oracle to answer questions.…”

“…Language-Active Environment Room-to-Room (Anderson et al, 2018b) Matterport3D Indoor Room-for-Room Matterport3D Indoor Room-Across-Room (Ku et al, 2020) Matterport3D Indoor Landmark-RxR (He et al, 2021) Matterport3D Indoor XL-R2R (Yan et al, 2020) Matterport3D Indoor VLNCE (Krantz et al, 2020) Habitat Indoor StreetLearn Google Street View Outdoor StreetNav (Hermann et al, 2020) Google Street View Outdoor TOUCHDOWN Google Street View Outdoor Talk2Nav (Vasudevan et al, 2021) Google Street View Outdoor LANI (Misra et al, 2018) -Outdoor RoomNav (Wu et al, 2018) House3D Indoor EmbodiedQA (Das et al, 2018) House3D Indoor REVERIE Matterport3D Indoor SOON (Zhu et al, 2021a) Matterport3D Indoor IQA (Gordon et al, 2018) AI2-THOR Indoor CHAI (Misra et al, 2018) CHALET Indoor ALFRED (Shridhar et al, 2020) AI2-THOR Indoor VNLA Matterport3D Indoor HANNA (Nguyen and Daumé III, 2019) Matterport3D Indoor CEREALBAR -Indoor Just Ask (Chi et al, 2020) Matterport3D Indoor CVDN (Thomason et al, 2019b) Matterport3D Indoor RobotSlang (Banerjee et al, 2020) -Indoor Talk the Walk (de Vries et al, 2018) -Outdoor MC Collab (Narayan- Minecraft Outdoor TEACh (Padmakumar et al, 2021) AI2-THOR Indoor DialFRED (Gao et al, 2022) AI2-THOR Indoor ) Habitat (Manolis Savva* et al, 2019 AI2-THOR Gibson (Xia et al, 2018) LANI (Misra et al, 2018) *Google Street View…”