Toward understanding natural language directions

Kollar, Thomas; Tellex, Stefanie; Roy, Deb; Roy, Nicholas

doi:10.1145/1734454.1734553

Cited by 116 publications

(128 citation statements)

References 5 publications

(5 reference statements)

Supporting

Mentioning

127

Contrasting

Order By: Relevance

“…Our approach extends previous research in understanding natural language directions [1] to infer three-dimensional paths. We implement this system on an autonomous MAV, so that a user can command the vehicle using phrases such as "Fly up to the windows" (Fig.…”

Section: Introductionsupporting

confidence: 52%

“…Our previous work building systems for understanding natural language directions modeled directions as a sequence of landmarks [6] while accounting for spatial relations and a limited set of verbs [1]. The notion of spatial description clauses is influenced by many similar formalisms for reasoning about the semantics of natural language directions [7], [8], [9], [10], [11], [12].…”

Section: Related Workmentioning

confidence: 99%

“…First, the commands are parsed into spatial description clauses (SDCs), semantic structures that robustly capture the meaning of spatial directions [1]. Next, a probabilistic direction understanding module uses the SDCs together with a map of the environment to infer the maximum likelihood path conveyed by the natural language directions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Natural language command of an autonomous micro-air vehicle

Huang

Tellex

Bachrach

et al. 2010

2010 IEEE/RSJ International Conference on Intelligent Robots and Systems

View full text Add to dashboard Cite

Abstract-Natural language is a flexible and intuitive modality for conveying directions and commands to a robot but presents a number of computational challenges. Diverse words and phrases must be mapped into structures that the robot can understand, and elements in those structures must be grounded in an uncertain environment.In this paper we present a micro-air vehicle (MAV) capable of following natural language directions through a previously mapped and labeled environment. We extend our previous work in understanding 2D natural language directions to three dimensions, accommodating new verb modifiers such as go up and go down, and commands such as turn around and face the windows. We demonstrate the robot following directions created by a human for another human, and interactively executing commands in the context of surveillance and search and rescue in confined spaces. In an informal study, 71% of the paths computed from directions given by one user terminated within 10 m of the desired destination.

show abstract

Section: Introductionsupporting

confidence: 52%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Natural language command of an autonomous micro-air vehicle

Huang

Tellex

Bachrach

et al. 2010

2010 IEEE/RSJ International Conference on Intelligent Robots and Systems

View full text Add to dashboard Cite

show abstract

“…Tellex et al [6] developed a probabilistic graphical model to infer object pick-and-place tasks for execution by a forklift robot from natural language commands. Kollar et al [7] employed a Bayesian approach for interpreting route directions on a mobile robot. In both of these works there was no explicit definition of the spatial relations used, static or otherwise, and instead they were learned from labeled training data.…”

Section: Related Workmentioning

confidence: 99%

Using Spatial Semantic and Pragmatic Fields to Interpret Natural Language Pick-and-Place Instructions for a Mobile Service Robot

Fasola

Matarić

2013

Social Robotics

View full text Add to dashboard Cite

Abstract. We present a methodology for enabling mobile service robots to follow natural language instructions for object pick-and-place tasks from nonexpert users, with and without user-specified constraints, and with a particular focus on spatial language understanding. Our approach is capable of addressing both the semantic and pragmatic properties of object movement-oriented natural language instructions, and in particular, proposes a novel computational field representation for the incorporation of spatial pragmatic constraints in mobile manipulation task planning. The design and implementation details of our methodology are also presented, including the grammar utilized and our procedure for pruning multiple candidate parses based on context. The paper concludes with an evaluation of our approach implemented on a simulated mobile robot operating in both 2D and 3D home environments.

show abstract

“…For example, consider a written instruction for following a particular route such as "Continue to walk straight, going through one door until you come to an intersection just past a whiteboard." (Kollar et al 2010). An interactive version of the same instruction from the corpus is substantially more complex:…”

Section: Human Performance In a Search Taskmentioning

confidence: 99%

Toward Humanlike Task‐Based Dialogue Processing for Human Robot Interaction

2011

View full text Add to dashboard Cite

ArticlesWINTER 2011 77 I nteractions in natural language dialogues are an essential part of human social exchanges, ranging from social conventions such as greetings, to simple question-answer pairs, to task-based dialogues for coordinating activities, topic-based discussions, and all kinds of more open-ended conversations. As a result, the ability of future social and service robots to interact with humans in natural ways ) will critically depend on developing capabilities of humanlike dialoguebased natural language processing (NLP) in robotic architectures. However, different from other NLP contexts such as story understanding or machine translation, natural language processing on robots has at least the following six properties: realtime, parallel, spoken, embodied, situated, and dialogue-based.Real-time means that all processing must occur within the time frame of human processing, both at the level of comprehension as well as production. It also means that constraints will have to be incorporated incrementally as they occur, analogous to human language processing.Parallel means that all stages of language processing must operate concurrently to mutually constrain possible meaning interpretations and to allow for the generation of responses (such as acknowledgements) while an ongoing utterance is being processed.Spoken means that language processing necessarily operates on imperfect acoustic signals with varying quality that depends on the speaker and the background noise. In addition to handling prosodic variations, this includes typical features of spontaneous speech such as various types of disfluencies, slips of the tongue, or other types of errors that are usually not found in written texts.Embodied means that robots have to be able to process multimodal linguistic cues such as deictic terms accompanied by bodily movements, or other gestures that constrain possible interpretations of linguistic expressions. It also means that the robot will have to be able to produce similar gestures that are expected by human interlocutors to accompany certain linguistic constructs.

show abstract

Toward understanding natural language directions

Cited by 116 publications

References 5 publications

Natural language command of an autonomous micro-air vehicle

Natural language command of an autonomous micro-air vehicle

Using Spatial Semantic and Pragmatic Fields to Interpret Natural Language Pick-and-Place Instructions for a Mobile Service Robot

Toward Humanlike Task‐Based Dialogue Processing for Human Robot Interaction

Contact Info

Product

Resources

About