RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Abstract

Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned modelbased planning framework is comparable to and sometimes better than human subjects on the tested tasks. 1

DefGraspNets: Grasp Planning on 3D Fields with Graph Neural Nets

“…Graph neural networks (GNNs) have been used to efficiently learn dynamics models for granular solids, deformable solids, and fluids [31][32][33][34][35]. Inspiring our work, MeshGraphNets [34] used GNNs to learn accurate dynamics for deformable solids using mesh-based representations, training from an industry-standard FEM solver.…”

“…It predicted deformation and stress on a 3D deformable plate with kinematically-actuated colliding shapes and achieved evaluation speeds up to two orders of magnitude faster than the solver. RoboCraft [35] used GNNs to learn how plasticine-like objects with particle representations deform under interaction with a robotic gripper, training from visual input. Whereas MeshGraphNets used forward passes through the networks to predict dynamics, RoboCraft also used backwards passes to perform gradient-based trajectory optimization, molding the plasticine into a desired shape.…”

Modal-graph 3D shape servoing of deformable objects with raw point clouds

“…They designed a sim-to-real method that both uses DNNs to extract features from point clouds and to learn manipulation laws to deform the object to a goal point cloud. Instead of extracting features from point clouds, Shi et al (2022) proposed a model-based method to shape elasto-plastic objects, which transforms RGB-D data into particles and learns deformation behaviors using particle-based graph neural networks (GNNs). The work (Shen et al (2022)) studying goal-conditioned DOM also involves deforming volumetric objects toward goal point clouds.…”

“…The aforementioned point-cloud-based works require additional processing of the noisy point clouds, such as nonrigid registrations (Jin et al (2019)), occlusion removal (Hu et al (2019)), re-samplings (Lagneau et al (2020a); Zhou et al (2021); Thach et al (2022)), correspondence identification (Shen et al (2022)), and surface reconstructions/refinements (Shi et al (2022)). In comparison, our method can directly extract deformation features from raw point clouds without extra point processing.…”

“…The reasons are twofold: first, it is hard to model the relationship between high-dimensional point clouds and robot manipulation; second, shape perceptions from raw point clouds suffer from their unknown point-wise correspondences and noisy partial observability. Existing works require extra processing (such as re-samplings (Lagneau et al (2020a)), registrations (Jin et al (2019)), and reconstructions (Shi et al (2022); Hu et al (2019))) to refine or recover the shape geometry from raw point clouds. However, from the perspective of modal analysis (Pentland and Sclaroff (1991)), irregularities and noises of local measurements mainly influence the high-frequency deformation modes, leaving the low-frequency modes relatively unchanged.…”