A model for verifiable grounding and execution of complex natural language instructions

Boteanu, Adrian; Howard, Thomas M.; Arkin, Jacob; Kress-Gazit, Hadas

doi:10.1109/iros.2016.7759412

Cited by 25 publications

(24 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The language grounding step was completed in 0.33 seconds for (a) and 0.37 seconds for (b); the inferred command and grounding triggered the motion planning and task execution routine that required 10 seconds in each case. Additional examples of physical demonstrations of natural language interfaces to robots based on variations of the DCG on robot torsos, unmanned ground vehicles, and assistive robotic manipulators are described in Boteanu et al (2016), Oh et al (2016), and Broad et al (2017).…”

Section: Resultsmentioning

confidence: 99%

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

Paul

Arkin

Aksaray

et al. 2018

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

Our goal is to develop models that allow a robot to efficiently understand or "ground" natural language instructions in the context of its world representation. Contemporary approaches estimate correspondences between language instructions and possible groundings such as objects, regions, and goals for actions that the robot should execute. However, these approaches typically reason in relatively small domains and do not model abstract spatial concepts such as as "rows, " "columns, " or "groups" of objects and, hence, are unable to interpret an instruction such as "pick up the middle block in the row of five blocks. " In this paper, we introduce two new models for efficient natural language understanding of robot instructions. The first model, which we call the adaptive distributed correspondence graph (ADCG), is a probabilistic model for interpreting abstract concepts that require hierarchical reasoning over constituent concrete entities as well as notions of cardinality and ordinality. Abstract grounding variables form a Markov boundary over concrete groundings, effectively de-correlating them from the remaining variables in the graph. This structure reduces the complexity of model training and inference. Inference in the model is posed as an approximate search procedure that orders factor computation such that the estimated probable concrete groundings focus the search for abstract concepts towards likely hypothesis, pruning away improbable portions of the exponentially large space of abstractions. Further, we address the issue of scalability to complex domains and introduce a hierarchical extension to a second model termed the hierarchical adaptive distributed correspondence graph (HADCG). The model utilizes the abstractions in the ADCG but infers a coarse symbolic structure from the utterance and the environment model and then performs fine-grained inference over the reduced graphical model, further improving the efficiency of inference. Empirical evaluation demonstrates accurate grounding of abstract concepts embedded in complex natural language instructions commanding a robotic torso and a mobile robot. Further, the proposed approximate inference method allows significant efficiency gains compared with the baseline, with minimal trade-off in accuracy.

show abstract

Section: Resultsmentioning

confidence: 99%

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

Paul

Arkin

Aksaray

et al. 2018

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…The question of how to effectively convert between natural language instructions and robot behavior has been widely studied in previous work [50,34,24,14,9,47,8,18,11,1,33,28,36,27,40,7,2,19,37]. So far, there have been three categories of behavior specifications that these works have mapped natural language to: action sequences, goal states, and LTL specifications.…”

Section: Related Workmentioning

confidence: 99%

“…Lignos et al [28] instead used off-the-shelf parsers to interpret the full space of natural language, but do not provide a method to learn from new robot-specific language, and therefore their approach may be limited by the particular corpus used by the existing parser. In contrast, Boteanu et al [7] collected a crowdsourced corpus of block-sorting instructions to train a Verifiable Distributed Correspondence Graph model that mapped natural language to structured English. In this work, we present a corpus that is an order of magnitude larger, and apply the sequence-to-sequence framework that allows grounding to the full space of LTL formulae (instead of the GR(1) fragment).…”

Section: Related Workmentioning

confidence: 99%

Sequence-to-Sequence Language Grounding of Non-Markovian Task Specifications

Gopalan

Arumugam

Wong

et al. 2018

Robotics: Science and Systems XIV

View full text Add to dashboard Cite

Abstract-Often times, natural language commands issued to robots not only specify a particular target configuration or goal state but also outline constraints on how the robot goes about its execution. That is, the path taken to achieving some goal state is given equal importance to the goal state itself. One example of this could be instructing a wheeled robot to "go to the living room but avoid the kitchen," in order to avoid scuffing the floor. This class of behaviors poses a serious obstacle to existing language understanding for robotics approaches that map to either action sequences or goal state representations. Due to the non-Markovian nature of the objective, approaches in the former category must map to potentially unbounded action sequences whereas approaches in the latter category would require folding the entirety of a robot's trajectory into a (traditionally Markovian) state representation, resulting in an intractable decision-making problem. To resolve this challenge, we use a recently introduced probabilistic variant of Linear Temporal Logic (LTL) as a goal specification language for a Markov Decision Process (MDP). While demonstrating that standard neural sequence-to-sequence learning models can successfully ground language to this semantic representation, we also provide analysis that highlights generalization to novel, unseen logical forms as an open problem for this class of model. We evaluate our system within two simulated robot domains as well as on a physical robot, demonstrating accurate language grounding alongside a significant expansion in the space of interpretable robot behaviors.

show abstract

“…Using natural language is the most efficient way to issue a command to robots, and since they have to operate in the physical world, understanding the way humans describe space is crucial. Current state-ofthe-art approaches to grounding natural language commands in general, and spatial commands in particular, are based on probabilistic graphical models (PGM) such as Generalized Grounding Graphs (G 3 ) (Tellex et al, 2011) and Distributed Correspondence Graphs (DCG) (Howard et al, 2014) and their modifications (Broad et al, 2016;Paul et al, 2016;Boteanu et al, 2016;Chung et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the First International Workshop on Spatial Language Understanding

2018

View full text Add to dashboard Cite

The challenge for computational models of spatial descriptions for situated dialogue systems is the integration of information from different modalities. The semantics of spatial descriptions are grounded in at least two sources of information: (i) a geometric representation of space and (ii) the functional interaction of related objects that. We train several neural language models on descriptions of scenes from a dataset of image captions and examine whether the functional or geometric bias of spatial descriptions reported in the literature is reflected in the estimated perplexity of these models. The results of these experiments have implications for the creation of models of spatial lexical semantics for human-robot dialogue systems. Furthermore, they also provide an insight into the kinds of the semantic knowledge captured by neural language models trained on spatial descriptions, which has implications for image captioning systems.

show abstract

A model for verifiable grounding and execution of complex natural language instructions

Cited by 25 publications

References 16 publications

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

Sequence-to-Sequence Language Grounding of Non-Markovian Task Specifications

Proceedings of the First International Workshop on Spatial Language Understanding

Contact Info

Product

Resources

About