Contextual awareness: Understanding monologic natural language instructions for autonomous robots

Arkin, Jacob; Walter, Matthew R.; Boteanu, Adrian; Napoli, Michael E.; Harel, Biggie,; Kress-Gazit, Hadas; Howard, Thomas M.

doi:10.1109/roman.2017.8172349

Cited by 11 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A well-studied problem in this setting is the Vision-and-language Navigation, where the tasks consist of navigating to a desired location in an environment, given a natural language command and an egocentric view from the agent's current position [2,9,36]. Another subcategory of instruction-following involves instructing an embodied agent using natural language [32,15,5,29,30,22,31]. Our proposed setting is different from instruction-following, in that the goal of the target task is not being communicated using language alone; instead, a demonstration for a related task (source task) is available, and language is used to communicate the difference between the demonstrated task and the target task.…”

Section: Related Workmentioning

confidence: 99%

Zero-shot Task Adaptation using Natural Language

Goyal,

Mooney,

Niekum

2021

Preprint

View full text Add to dashboard Cite

Imitation learning and instruction-following are two common approaches to communicate a user's intent to a learning agent. However, as the complexity of tasks grows, it may be beneficial to use both demonstrations and language to communicate with an agent. In this work, we propose a novel setting where, given a demonstration for a task (the source task), and a natural language description of the differences between the demonstrated task and a related but different task (the target task), our goal is to train an agent to complete the target task in a zero-shot setting-that is, without any demonstrations for the target task. To this end, we introduce Language-Aided Reward and Value Adaptation (LARVA) which, given a source demonstration and a linguistic description of how the target task differs, learns to output either a reward or value function that accurately reflects the target task. Our experiments show that on a diverse set of adaptations, our approach is able to complete more than 95% of target tasks when using template-based descriptions, and more than 70% when using free-form natural language.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Zero-shot Task Adaptation using Natural Language

Goyal,

Mooney,

Niekum

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Understanding sequences of natural language utterances has been addressed using semantic parsing (e.g., Miller et al, 1996;MacMahon et al, 2006;Chen and Mooney, 2011;Artzi and Zettlemoyer, 2013;Artzi et al, 2014;Long et al, 2016;Iyyer et al, 2017;Suhr et al, 2018;Arkin et al, 2017;Broad et al, 2017). Interactions were also used for semantic parser induction (Artzi and Zettlemoyer, 2011;Thomason et al, 2015;Wang et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Executing Instructions in Situated Collaborative Interactions

Suhr

Yan²,

Schluger

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.

show abstract

“…More recently, Paul et al proposed a method to deal with abstract spatial concepts in instructions [10]. Arkin et al proposed a variant of a contemporary probabilistic graphical model for language understanding in a context [11].…”

Section: Related Workmentioning

confidence: 99%

“…In prior studies, Paul et al and Arkin et al focused on directly grounding natural language onto the phenomena and objects in the real world [10], [11]. We extend SDC to make more accurate for grounding natural language onto the real-world, and to accept various expressions in verbal instructions for parking a car.…”

Section: Extended Sdcmentioning

confidence: 99%

Parsing Parking Instructions for Self-driving Cars into Spatial Semantic Descriptions

Inago¹

2019

JCP

View full text Add to dashboard Cite

This study is motivated by an attempt to develop a system that can control a self-driving car by words. To achieve this, the verbal instructions we provide to the car must be grounded onto the real-world phenomena. With this background, in this study, we extend the framework of semantic description clause (SDC) proposed by Kollar et al. (2010) by adding two new semantic categories, VIEW and STATE, so as to be able to ground more variety of the instructions for driving a car in real-world environment. We developed a pipeline of a CCG parser, a reranker of parse trees and a converter from parse trees into SDCs. The result of parsing the instructions with the extended CCG grammar shows 84.7% accuracy on 6,019 various Japanese driving instructions. We finally convert the parse trees into SDCs using conversion rules, resulting in 90.4% accuracy on 5,638 parse trees.

show abstract

Contextual awareness: Understanding monologic natural language instructions for autonomous robots

Cited by 11 publications

References 22 publications

Zero-shot Task Adaptation using Natural Language

Zero-shot Task Adaptation using Natural Language

Executing Instructions in Situated Collaborative Interactions

Parsing Parking Instructions for Self-driving Cars into Spatial Semantic Descriptions

Contact Info

Product

Resources

About