Leveraging Large-Scale Semantic Networks for Adaptive Robot Task Learning and Execution

Abstract

This work seeks to leverage semantic networks containing millions of entries encoding assertions of commonsense knowledge to enable improvements in robot task execution and learning. The specific application we explore in this project is object substitution in the context of task adaptation. Humans easily adapt their plans to compensate for missing items in day-to-day tasks, substituting a wrap for bread when making a sandwich, or stirring pasta with a fork when out of spoons. Robot plan execution, however, is far less robust, with missing objects typically leading to failure if the robot is not aware of alternatives. In this article, we contribute a context-aware algorithm that leverages the linguistic information embedded in the task description to identify candidate substitution objects without reliance on explicit object affordance information. Specifically, we show that the task context provided by the task labels within the action structure of a task plan can be leveraged to disambiguate information within a noisy large-scale semantic network containing hundreds of potential object candidates to identify successful object substitutions with high accuracy. We present two extensive evaluations of our work on both abstract and real-world robot tasks, showing that the substitutions made by our system are valid, accepted by users, and lead to a statistically significant reduction in robot learning time. In addition, we report the outcomes of testing our approach with a large number of crowd workers interacting with a robot in real time.

From Multi-Modal Property Dataset to Robot-Centric Conceptual Knowledge About Household Objects

“…In this experiment, we have pitted our conceptual knowledge against WordNet in the substitute selection scenario. Our objective is to demonstrate that common sense knowledge bases such as WordNet are not adequate for substitute selection without selecting suitable knowledge a priori as done in Boteanu et al (2016).…”

“…In the literature on substitute selection, typically a substitute for a missing tool is determined by means of knowledge about object, and the knowledge-driven similarity between a missing tool prototype and a potential substitute. Such knowledge about objects varies in its contents and form across the literature: metric data about position, orientation, size, and symbolic knowledge about handpicked relations such as similar-to and capable-of extracted from ConceptNet (Bansal et al, 2020); visual and physical understanding of multi-object interactions demonstrated by humans (Xie et al, 2019); matching similarity of shapes of point clouds and materials based on the spectrometer data using dual neural network (Shrivatsav et al, 2019); metric data about size, shape and grasp, as well as a human estimate of an affordance score for task + mass (Abelha and Guerin, 2017); attributes and affordances of objects are hand-coded using a logic-based notation, and a multidimensional conceptual space of features such as shape and color intensity (Mustafa et al, 2016); hand-coded models of known tools in terms of superquadrics and relationships among them (Abelha et al, 2016); potential candidates extracted from WordNet and ConceptNet if they share the same parent with a missing tool for predetermined relations: has-property, capable-of, and used-for (Boteanu et al, 2016); hand-coded object-action relations (Agostini et al, 2015); as well as hand-coded knowledge about inheritance and equivalence relations among objects and affordances (Awaad et al, 2014). While for tool selection, metric data of certain properties are primarily considered, for substitute selection, symbolic knowledge about the object category or class is considered.…”

Towards Robust One-shot Task Execution using Knowledge Graph Embeddings

“…Most closely related to our approach are [3] and [2], which leverage different inputs and computational frameworks to generalize task plans for new execution environments in addition to having differing assumptions. In [3] online ontologies and repositories of annotated task demonstrations are provided as inputs to learn a probabilistic graphical model, a Markov Logic Network (MLN).…”

“…Generalizations can include alternate primitive actions and primitive action parameters. In [2], a random forest classifier is trained on similarity features computed over general lexical databases of objects and object attributes provided as an input. The classifier is then used to infer valid object substitutions for objects used in a demonstrated task plan and assumes the same primitive action is being performed.…”

“…However, task plans that are learned in domains rich with semantic knowledge can benefit from incorporating such domain knowledge into the task plan. Task plan constituents from a prototypical or demonstrated task plan can be effectively generalized to new execution environments by leveraging statistical correlations or heuristic features learned from the task domain [2], [3].…”