IFOR: Iterative Flow Minimization for Robotic Object Rearrangement

Abstract

Figure 1. An example of IFOR being applied to real data. The initial and goal scenes are shown on the left.Our approach allows the robot to repeatedly identify transformations that will minimize the flow for various objects between the current and goal scenes. It can then repeatedly grasp, move, and place objects, rotating as necessary, in order to achieve the configuration in the goal scene. The system is trained completely on synthetic data and transfers to the real world in zero-shot manner.

IFOR: Iterative Flow Minimization for Robotic Object Rearrangement

“…The use of depth input has also been extensively studied. Methods like CLIPort [3] and IFOR [1] directly process the RGB-D images for object manipulation, and hence are limited to simple pickand-place tasks in 2D top-down settings. To overcome this issue, explicit 3D representations such as point clouds have been utilized.…”

“…Learning a single model for many different tasks has been of particular interest to the robotics community recently. A large volume of work achieves the multi-task generalization by using a generalizable task or action representation such as object point cloud [18,19], semantic segmentation and optical flow [1], and object-centric representation [29,30]. However, the limited expressiveness of such representations constrains them to only generalize within a task category.…”

“…More visualizations of the task setups and the model performance are also provided. 1 Ablation Study. We conduct ablation experiments to analyze different design choices of RVT: (a) the resolution of the rendered images ("Im.…”

“…A popular class of learning methods directly processes image(s) viewed from single or multiple cameras. These view-based methods have achieved impressive success on a variety of pick-and-place and object rearrangement tasks [1,2,3,4]. However, their success on tasks that require 3D reasoning has been limited.…”

StructDiffusion: Object-Centric Diffusion for Semantic Rearrangement of Novel Objects

“…In our approach, we use unknown object instance segmentation to break our scene up into objects, as per prior work [6], [7], [8], [9]. Then, we use a multi-modal transformer to combine both word tokens and object encodings from Point Cloud Transformer [10] in order to make 6-DoF goal pose predictions.…”