Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight

Abstract

Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects. However, learning a model that can both perform complex tasks and generalize to previously unseen objects and goals remains a significant challenge. We study this challenge in the context of "improvisational" tool use: a robot is presented with novel objects and a user-specified goal (e.g., sweep some clutter into the dustpan), and must figure out, using only raw image observations, how to accomplish the goal using the available objects as tools. We approach this problem by training a model with both a visual and physical understanding of multi-object interactions, and develop a sampling-based optimizer that can leverage these interactions to accomplish tasks. We do so by combining diverse demonstration data with self-supervised interaction data, aiming to leverage the interaction data to build generalizable models and the demonstration data to guide the model-based RL planner to solve complex tasks. Our experiments show that our approach can solve a variety of complex tool use tasks from raw pixel inputs, outperforming both imitation learning and self-supervised learning individually. Furthermore, we show that the robot can perceive and use novel objects as tools, including objects that are not conventional tools, while also choosing dynamically to use or not use tools depending on whether or not they are required. Videos of the results are available online 1 .

How to train your robot with deep reinforcement learning: lessons we have learned

“…7. Using both unsupervised interaction and teleoperated demonstration data, the robot learns a visual dynamics model and action proposal model that enables it to perform new tasks with novel, previously unseen tools (using Xie et al, 2019). The task specification is shown on the left and the robot performing the task is shown on the right.…”

Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning

“…Similarly, while artificial intelligence (AI) systems have become increasingly adept at perceiving and manipulating objects, none perform the sort of rapid mechanical reasoning that people do. Some artificial agents learn to use tools from expert demonstrations (7), which limits their flexibility. Others learn from thousands of years of simulated experience (8), which is significantly longer than required for people.…”

From Multi-Modal Property Dataset to Robot-Centric Conceptual Knowledge About Household Objects

“…In the literature on substitute selection, typically a substitute for a missing tool is determined by means of knowledge about object, and the knowledge-driven similarity between a missing tool prototype and a potential substitute. Such knowledge about objects varies in its contents and form across the literature: metric data about position, orientation, size, and symbolic knowledge about handpicked relations such as similar-to and capable-of extracted from ConceptNet (Bansal et al, 2020); visual and physical understanding of multi-object interactions demonstrated by humans (Xie et al, 2019); matching similarity of shapes of point clouds and materials based on the spectrometer data using dual neural network (Shrivatsav et al, 2019); metric data about size, shape and grasp, as well as a human estimate of an affordance score for task + mass (Abelha and Guerin, 2017); attributes and affordances of objects are hand-coded using a logic-based notation, and a multidimensional conceptual space of features such as shape and color intensity (Mustafa et al, 2016); hand-coded models of known tools in terms of superquadrics and relationships among them (Abelha et al, 2016); potential candidates extracted from WordNet and ConceptNet if they share the same parent with a missing tool for predetermined relations: has-property, capable-of, and used-for (Boteanu et al, 2016); hand-coded object-action relations (Agostini et al, 2015); as well as hand-coded knowledge about inheritance and equivalence relations among objects and affordances (Awaad et al, 2014). While for tool selection, metric data of certain properties are primarily considered, for substitute selection, symbolic knowledge about the object category or class is considered.…”