RoboBrain: Large-Scale Knowledge Engine for Robots

Abstract

In this paper we introduce a knowledge engine, which learns and shares knowledge representations, for robots to carry out a variety of tasks. Building such an engine brings with it the challenge of dealing with multiple data modalities including symbols, natural language, haptic senses, robot trajectories, visual features and many others. The knowledge stored in the engine comes from multiple sources including physical interactions that robots have while performing tasks (perception, planning and control), knowledge bases from the Internet and learned representations from several robotics research groups.We discuss various technical aspects and associated challenges such as modeling the correctness of knowledge, inferring latent information and formulating different robotic tasks as queries to the knowledge engine. We describe the system architecture and how it supports different mechanisms for users and robots to interact with the engine. Finally, we demonstrate its use in three important research areas: grounding natural language, perception, and planning, which are the key building blocks for many robotic tasks. This knowledge engine is a collaborative effort and we call it RoboBrain.

Towards Robust One-shot Task Execution using Knowledge Graph Embeddings

“…Prior work in open-world planning has developed techniques to make robots programmed by expert users more robust to environment variation through the use of commonsense or domain knowledge [19], [20], [21], [22], [23]. Open-world planners can achieve plan goals in environments with variations [19], under-specified plans [21], [24], and partially observed worlds [20] by coupling a knowledge base with declarative programming planners, such as PDDL [19], CRAM [21], or ASP [20].…”

“…Our domain knowledge representation uses an explicit model of world semantics in the form of a knowledge graph G composed of individual facts or triples (h, r, t) with h and t being the head and tail entities (respectively) for which the relation r holds, e.g. (cup, hasAction, fill) [26], [23], [22], [27].…”

Continual Learning of Knowledge Graph Embeddings

“…Modeling Semantic Knowledge in Robotics methods commonly use an explicit model of world semantics in the form of a knowledge graph G composed of individual facts or triples (h, r, t) with h and t being the head and tail entities (respectively) for which the relation r holds, e.g. (cup, hasAction, fill) [21], [22], [23], [24]. Recent work has modeled G using distributed representations because of their ability to approximate proximity of meaning from vector computations [1], [2], [3], [4], [5], [6].…”

Laser2Vec: Similarity-based Retrieval for Robotic Perception Data

“…Thus, most robotic systems of the future will maintain associated databases storing past sensor data. The nascent interdisciplinary work between robotics and databases includes knowledge engines [25], failure provenance [22], and notably, Vroom [19], a project with similar ambitions. To handle robotic perception data, Vroom memoizes the incoming data using pre-existing classifiers, typically done onboard the robot.…”