Approaching the Symbol Grounding Problem with Probabilistic Graphical Models

Tellex, Stefanie; Kollar, Thomas; Dickerson, Steven; Walter, Matthew R.; Banerjee, Ashis G.; Teller, Seth; Roy, Nicholas

doi:10.1609/aimag.v32i4.2384

Cited by 115 publications

(108 citation statements)

References 27 publications

(49 reference statements)

Supporting

Mentioning

101

Contrasting

Order By: Relevance

“…In order to derive an expression for the robot's uncertainty about groundings in the external world, our approach builds on the Generalized Grounding Graph (G 3 ) framework [18,17]. The G 3 framework defines a probabilistic model that maps between parts of the language and groundings in the external world, which can be objects, places, paths, or events.…”

Section: (A)mentioning

confidence: 99%

Toward Information Theoretic Human-Robot Dialog

Tellex¹,

Thaker²,

Deits³

et al. 2012

Robotics: Science and Systems VIII

Self Cite

View full text Add to dashboard Cite

Abstract-Our goal is to build robots that can robustly interact with humans using natural language. This problem is challenging because human language is filled with ambiguity, and furthermore, due to limitations in sensing, the robot's perception of its environment might be much more limited than that of its human partner. To enable a robot to recover from a failure to understand a natural language utterance, this paper describes an information-theoretic strategy for asking targeted clarifying questions and using information from the answer to disambiguate the language. To identify good questions, we derive an estimate of the robot's uncertainty about the mapping between specific phrases in the language and aspects of the external world. This metric enables the robot to ask a targeted question about the parts of the language for which it is most uncertain. After receiving an answer, the robot fuses information from the command, the question, and the answer in a joint probabilistic graphical model in the G 3 framework. When using answers to questions, we show the robot is able to infer mappings between parts of the language and concrete object groundings in the external world with higher accuracy than by using information from the command alone. Furthermore, we demonstrate that by effectively selecting which questions to ask, the robot is able to achieve significant performance gains while asking many fewer questions than baseline metrics.

show abstract

Section: (A)mentioning

confidence: 99%

Toward Information Theoretic Human-Robot Dialog

Tellex¹,

Thaker²,

Deits³

et al. 2012

Robotics: Science and Systems VIII

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, such a task cannot be solved using static visual object recognition methods as detecting whether an object is full or empty may often require the robot to perform a certain action on it (e.g., lift the object to measure the force it exerts on the arm). In this section, the research contribution focuses on solving the symbol grounding problem (Harnad 1990), a longstanding challenge in AI, where language is grounded using the robot's perception and action (Tellex et al 2011;Matuszek et al 2012;Krishnamurthy and Kollar 2013;Perera and Allen 2013;Kollar, Krishnamurthy and Strimel 2013;Tellex et al 2014;Matuszek et al 2014;Parde et al 2015;Spranger and Steels 2015). To address this problem, we enable a robot to undergo two distinct developmental stages:…”

Section: Grounded Language Learning Through Human-robot Interactionmentioning

confidence: 99%

BWIBots: A platform for bridging the gap between AI and human–robot interaction research

Khandelwal

Zhang

Sinapov

et al. 2017

The International Journal of Robotics Research

View full text Add to dashboard Cite

Recent progress in both Artificial Intelligence (AI) and Robotics have enabled the development of general purpose robot platforms that are capable of executing a wide variety of complex, temporally extended service tasks in open environments. This article introduces a novel, custom-designed multi-robot platform for research on AI, robotics, and especially Human-Robot Interaction (HRI) for service robots. Called BWIBots, the robots were designed as a part of the Building-Wide Intelligence (BWI) project at the University of Texas at Austin. The article begins with a description of, and justification for, the hardware and software design decisions underlying the BWIBots, with the aim of informing the design of such platforms in the future. It then proceeds to present an overview of various research contributions that have enabled the BWIBots to better (i) execute action sequences to complete user requests, (ii) efficiently ask questions to resolve user requests, (iii) understand human commands given in natural language, and (iv) understand human intention from afar. The article concludes with a look forward towards future research opportunities and applications enabled by the BWIBot platform.

show abstract

“…Tellex et al [6] developed a probabilistic graphical model to infer object pick-and-place tasks for execution by a forklift robot from natural language commands. Kollar et al [7] employed a Bayesian approach for interpreting route directions on a mobile robot.…”

Section: Related Workmentioning

confidence: 99%

Using Spatial Semantic and Pragmatic Fields to Interpret Natural Language Pick-and-Place Instructions for a Mobile Service Robot

Fasola

Matarić

2013

Social Robotics

View full text Add to dashboard Cite

Abstract. We present a methodology for enabling mobile service robots to follow natural language instructions for object pick-and-place tasks from nonexpert users, with and without user-specified constraints, and with a particular focus on spatial language understanding. Our approach is capable of addressing both the semantic and pragmatic properties of object movement-oriented natural language instructions, and in particular, proposes a novel computational field representation for the incorporation of spatial pragmatic constraints in mobile manipulation task planning. The design and implementation details of our methodology are also presented, including the grammar utilized and our procedure for pruning multiple candidate parses based on context. The paper concludes with an evaluation of our approach implemented on a simulated mobile robot operating in both 2D and 3D home environments.

show abstract

Approaching the Symbol Grounding Problem with Probabilistic Graphical Models

Cited by 115 publications

References 27 publications

Toward Information Theoretic Human-Robot Dialog

Toward Information Theoretic Human-Robot Dialog

BWIBots: A platform for bridging the gap between AI and human–robot interaction research

Using Spatial Semantic and Pragmatic Fields to Interpret Natural Language Pick-and-Place Instructions for a Mobile Service Robot

Contact Info

Product

Resources

About