StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Abstract

Rearrange objects that are smaller than the green glass pan tower, top, left, west line, top, left, large circle, top, right, large, north Rearrange objects that have the same color as the glass stapler tower, top, right, west line, bottom, left, large circle, top, left, small, west Rearrange yellow objects circle, bottom, right, medium circle, top, middle, medium circle, top, middle, large Rearrange yellow objects Rearrange objects that have the same material as the blue object circle, bottom, right, large circle, bottom, middle, medium circle, bottom, middle, large Set the table Rearrange objects that are smaller than the green glass pan tower, top, left, west line, top, left, large circle, top, right, large, north Rearrange objects that have the same color as the glass stapler tower, top, right, west line, bottom, left, large circle, top, left, small, west Rearrange yellow objects circle, bottom, right, medium circle, top, middle, medium circle, top, middle, large Rearrange yellow objects Rearrange objects that have the same material as the blue object circle, bottom, right, large circle, bottom, middle, medium circle, bottom, middle, large Set the table Put objects in a line Set the table Put objects in a line Make a circle

IFOR: Iterative Flow Minimization for Robotic Object Rearrangement

“…However, setting up a real-world TAMP system often requires substantial task-specific knowledge and accurate 3D models of the environment, significantly limiting the environments to which the system can generalize. To address this challenge, recent work has adopted deep learning-based approaches for robotic manipulation, for instance, on grasp planning [44,47,48,62,65], motion planning [7,57], and reasoning about spatial relations [20,36,49].…”

“…In contrast, we propose to use optical flow as the low-level feature descriptors, which can be naturally used to infer the full 6D transformations. In parallel to our work, recent efforts have also addressed rearrangement particularly learned from human demonstrations [14,72] and also with different goal specifications such as language [36,55].…”

“…With varying task setups, the desired goal state can be provided in different forms, for instance, a compact state representation [32,67] or natural language descriptions [36,55]. In this work, we address the rearrangement task where the goal state is specified by an RGB-D image [34,51], as shown in Fig.…”

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

“…Language-Instructed Manipulation Recently, various manipulation tasks have been researched with language input either describing the entire task, or serving interactive input for task specifications. Structformer [23] proposes an object selection network from language and visual encodings, as well as a language conditioned pose generator for semantic object rearrangement. Stepputtis et al [24] proposed a closed-loop control model for pouring tasks.…”

CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-horizon Robot Manipulation Tasks

“…Natural language processing has recently received much attention in the field of robotics [8], following the advances made towards learning groundings between vision and language [9], [10], [11]. Recent successes in humanrobot interaction include an interactive fetching system to localize objects mentioned in referring expressions [12], [13], [14], [15], [16] or grounding not only objects, but also spatial relations to follow language expressions characterizing pick-and-place commands [17], [18], [19]. By contrast, CALVIN tasks require grounding language to a wide variety of general-purpose robot skills.…”