Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

Abstract

Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the AL-FRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-toend architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT 1 (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.

A Modular Vision Language Navigation and Manipulation Framework for Long Horizon Compositional Tasks in Indoor Environment

“…In “MoViLan + PerfectMap”, ground truth BEV maps are provided to the agent as an ablation study for removal of our mapping module. Our framework demonstrates superior performance compared to the baseline ( Shridhar et al, 2020 ) algorithms, Moca ( Singh et al, 2020 ), HiTUT ( Zhang and Chai, 2021 ), HLSM ( Blukis et al, 2022 ), and LWIT ( Nguyen et al, 2021 ) on complete tasks. For sub-goal tasks, our framework has significantly higher path weighted success rates for “GoTo” compared to previous works (language instructions requiring pure navigation) because of novel mapping module, and hence higher overall success rates due to better positioning.…”

“…The VPM module in this study executes the interaction mask of the target object, and the APM module predicts the action sequence. HiTUT method ( Zhang and Chai, 2021 ) tries to increase the success rate of the ALFRED dataset by decomposing task learning into three sub tasks; sub-goal planning, scene navigation and object manipulation. All three sub tasks share the similar input form; therefore they solve together by applying an unified model upon on multi-task learning.…”

LLM-driven Instruction Following: Progresses and Concerns

“…• Learning from Explanations with Neural Execution Tree (Wang et al, 2020) • Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach (Yin et al, 2019) • Textual Entailment for Event Argument Extraction: Zero-and Few-Shot with Multi-Source Learning (Sainz et al, 2022) • Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (Liu et al, 2021) • True Few-Shot Learning With Prompts-A Real-World Perspective (Schick and Schütze, 2022) • The Turking Test: Can Language Models Understand Instructions? (Efrat and Levy, 2020) • Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (Zhang and Chai, 2021) • Cross-Task Generalization via Natural Language Crowdsourcing Instructions (Mishra et al, 2022) • MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction Following (Lou et al, 2023)…”

tagE: Enabling an Embodied Agent to Understand Human Instructions

“…Several approaches have been proposed to solve this. Predominant methods exploit the embodied nature of a robotic agent to infer and refine the plan by primarily using multi-modal input that includes visual feedback and action priors (Paxton et al, 2019;Shridhar et al, 2020;Zhang and Chai, 2021;Ahn et al, 2022). Thus natural language understanding in these systems is simplified by obtaining a latent representation of the language input to bias the inference using attention modeling.…”