Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions

Abstract

The goal of this article is to enable robots to perform robust task execution following human instructions in partially observable environments. A robot’s ability to interpret and execute commands is fundamentally tied to its semantic world knowledge. Commonly, robots use exteroceptive sensors, such as cameras or LiDAR, to detect entities in the workspace and infer their visual properties and spatial relationships. However, semantic world properties are often visually imperceptible. We posit the use of non-exteroceptive modalities including physical proprioception, factual descriptions, and domain knowledge as mechanisms for inferring semantic properties of objects. We introduce a probabilistic model that fuses linguistic knowledge with visual and haptic observations into a cumulative belief over latent world attributes to infer the meaning of instructions and execute the instructed tasks in a manner robust to erroneous, noisy, or contradictory evidence. In addition, we provide a method that allows the robot to communicate knowledge dissonance back to the human as a means of correcting errors in the operator’s world model. Finally, we propose an efficient framework that anticipates possible linguistic interactions and infers the associated groundings for the current world state, thereby bootstrapping both language understanding and generation. We present experiments on manipulators for tasks that require inference over partially observed semantic properties, and evaluate our framework’s ability to exploit expressed information and knowledge bases to facilitate convergence, and generate statements to correct declared facts that were observed to be inconsistent with the robot’s estimate of object properties.

A Review of NASA Human-Robot Interaction in Space

“…Efficient and accurate interpretation of instructions is particularly important for space missions involving robotic partners where communication or interaction between humans and robots is intermittent and bandwidth limited since the robot may not always have the ability to request a clarification when performing the task. Arkin et al have developed transparent and computationally efficient models for verifiable grounded language communication [54].…”

Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

“…Bayesian logic networks have been used to cope with noise and non-deterministic data from different data sources [13]. More recently, knowledge graph (KG) embedding models were introduced as scalable frameworks to model object knowledge encoded in multi-relational KGs [16,4]. Although the above techniques effectively model objects, they only support reasoning about binary class-level facts, therefore lacking the discriminative features needed to model object semantics in realistic environments.…”

“…In this paper, we compare to two variants of TuckER. The regular TuckER model follows existing work [16,4] to model binary relations between object class and object properties. • TuckER+ is a TuckER embedding model we implement to model binary relations between all pairs of property types (e.g., color and material, shape and location); it approximates an n-ary relation with a combination of binary relations.…”

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

“…To address this problem, several lines of research have shown that incorporating a variety of sensory modalities is the key to further enhance the robotic capabilities in recognizing multisensory object properties (see [4] and [21] for a review). For example, visual and physical interaction data yields more accurate haptic classification for objects [11], and non-visual sensory modalities (e.g., audio, haptics) coupled with exploratory actions (e.g., touch or grasp) have been shown useful for recognizing objects and their properties [5,10,15,24,30], as well as grounding natural language descriptors that people use to refer to objects [3,39]. More recently, researchers have developed end-to-end systems to enable robots to learn to perceive the environment and perform actions at the same time [20,42].…”