Grounding natural language instructions to semantic goal representations for abstraction and generalization

Arumugam, Dilip; Karamcheti, Siddharth; Gopalan, Nakul; Williams, Edward; Rhee, Mina; Wong, Lawson L. S.; Tellex, Stefanie

doi:10.1007/s10514-018-9792-8

Cited by 15 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tellex et al have developed methods to increase the speed and accuracy of inferring human intention from natural language. These methods are intended to increase the number of robots that a single astronaut could supervise [53].…”

Section: Human-robot Communicationmentioning

confidence: 99%

A Review of NASA Human-Robot Interaction in Space

2021

View full text Add to dashboard Cite

Purpose of Review This review provides an overview of the motivation, challenges, state-of-the-art, and recent research for human-robot interaction (HRI) in space. For context, we focus on NASA space missions, use cases, and systems (both flight and research). However, the discussion is broadly applicable to all activities in space that require or make use of human-robot teams. Recent Findings To date, HRI in space has largely been limited to remote interaction between humans on Earth and robots in space. This interaction is associated with telerobotic operations-from direct (manual) control to intermittent, supervisory control. Recent work, however, has begun to address a wide range of human-robot arrangements (co-located, remote, 1:1, groups, etc.). In addition, researchers have been studying human-robot teaming theory and system design, efficient interaction methods, and human-robot communication. Summary We begin by describing NASA's use of robots in space for both deep space science and human exploration. We then describe several aspects of HRI that are important for space missions, with emphasis on factors that are critical or unique for the space environment. Next, we provide a brief overview of HRI associated with space systems, including technology demonstrations. Finally, we conclude with a short survey of recent research, which will affect human-robot interaction for both Artemis missions and future missions to Mars.

show abstract

Section: Human-robot Communicationmentioning

confidence: 99%

A Review of NASA Human-Robot Interaction in Space

2021

View full text Add to dashboard Cite

show abstract

“…Vision & Dialogue Navigation and Task Completion Agents that additionally engage in dialogue can be learned by combining individual rule-based or learned components (Tellex et al 2016;Arumugam et al 2018;Thomason et al 2020). End-to-end VLN models can be improved by simulated clarification (Chi et al 2020;Nguyen and Daumé III 2019) and incorporating human-human conversation history (Thomason et al 2019;Zhu et al 2020).…”

Section: Related Workmentioning

confidence: 99%

TEACh: Task-Driven Embodied Agents That Chat

Padmakumar

Thomason

Shrivastava

et al. 2022

AAAI

View full text Add to dashboard Cite

Robots operating in human spaces must be able to engage in natural language interaction, both understanding and executing instructions, and using conversation to resolve ambiguity and correct mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human-human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates in natural language with a Follower. The Follower navigates through and interacts with the environment to complete tasks varying in complexity from "Make Coffee" to "Prepare Breakfast", asking questions and getting additional information from the Commander. We propose three benchmarks using TEACh to study embodied intelligence challenges, and we evaluate initial models' abilities in dialogue understanding, language grounding, and task execution.

show abstract

“…Several prior works have attempted to ground language with tasks or use language as a source of instructions for learning tasks with varying degrees of success ((MacMahon et al, 2006;Wang et al, 2016;Arumugam et al, 2017;Oh et al, 2017;Arumugam et al, 2019)). (Luketina et al, 2019) is a good reference for works combining language with sequential-decision making.…”

Section: Language Groundingmentioning

confidence: 99%

LISA: Learning Interpretable Skill Abstractions from Language

Garg¹,

Skanda²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning policies that effectually utilize language instructions in complex, multi-task environments is an important problem in imitation learning. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. To encode complex instructions into skills that can generalize to unseen instructions, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable skills from languageconditioned demonstrations. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation environments, LISA is able to outperform a strong non-hierarchical baseline in the low data regime and compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.

show abstract

Grounding natural language instructions to semantic goal representations for abstraction and generalization

Cited by 15 publications

References 20 publications

A Review of NASA Human-Robot Interaction in Space

A Review of NASA Human-Robot Interaction in Space

TEACh: Task-Driven Embodied Agents That Chat

LISA: Learning Interpretable Skill Abstractions from Language

Contact Info

Product

Resources

About