VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

Abstract

Robotic fabric manipulation has applications in cloth and cable management, senior care, surgery and more. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We address this problem by extending the recently proposed Visual Foresight framework to learn fabric dynamics, which can be efficiently reused to accomplish a variety of different fabric manipulation tasks with a single goalconditioned policy. We introduce VisuoSpatial Foresight (VSF), which extends prior work by learning visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. We experimentally evaluate VSF on multi-step fabric smoothing and folding tasks both in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time. Furthermore, we find that leveraging depth significantly improves performance for cloth manipulation tasks, and results suggest that leveraging RGBD data for video prediction and planning yields an 80% improvement in fabric folding success rate over pure RGB data. Supplementary material is available at https://sites.google.com/ view/fabric-vsf/.

Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies

“…Deformable object manipulation has seen a surge of interest recently in the robotics research community [18,34,33,28,6,10,25,26], but modeling state and action spaces in such tasks remains challenging. Much prior work focuses on the complex task of generating plans for manipulating cables and higher-dimensional deformable objects.…”

“…Other methods learn visuomotor policies end-to-end. Imitation learning [25], video prediction models [10], model-free reinforcement learning (RL) [17], and model-based RL [20] have successfully been applied to goal-conditioned fabric manipulation. In other end-to-end approaches, Nair et al [21] learn a dynamics model for performing template shape matching and knot-tying with rope.…”

“…Several recent works have also explored learning robot policies for deformable manipulation entirely in simulation and transferring them to the real world via domain randomization [26,28,34,6,10,26,35]. By contrast, we train perception systems using a combination of data from simulation and the physical world.…”

Disruption-Resistant Deformable Object Manipulation on Basis of Online Shape Estimation and Prediction-Driven Trajectory Correction

“…We have proposed a system for generating multi-step manipulation plans with variable goals [18][19] using forward models of the cloth dynamics. Subsequently, Kawaharazuka et al [20], Hoque et al [21] and Yan et al [22] proposed forward modelbased approaches capable of accommodating variable goal states to varying extents.…”

Parareal with a learned coarse model for robotic manipulation

“…The robot must complete the task without pushing other objects off the table or into the goal region to contact-based/non-prehensile manipulation planning and control-especially during re-planning or model-predictive control (MPC) where a robot executes an action in the realworld, gets the resulting state and then has to generate a new physics-based plan. Such MPC methods have been used in prior work to achieve manipulation robustness to parameter uncertainty [1], stabilize complex humanoid behaviours [35], and visually manipulate fabric [19].…”