VisuoSpatial Foresight for physical sequential fabric manipulation

Hoque, Ryan; Seita, Daniel; Balakrishna, Ashwin; Ganapathi, Aditya; Tanwani, Ajay Kumar; Jamali, Nawid; Yamane, Katsu; Iba, Soshi; Goldberg, Ken

doi:10.1007/s10514-021-10001-0

Cited by 30 publications

(65 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Secondly, primitive actions like grasping for volumetric deformable object remains a challenging problem [93]. Nonetheless, we believe recent advances in sim2real transfer methods for 1D and 2D deformable objects [33], [94], [95] could offer insight into closing the reality gap for volumetric deformable objects.…”

Section: Discussion Of Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation

Shen¹,

Jiang²,

Choy³

et al. 2022

Preprint

View full text Add to dashboard Cite

Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability. We introduce ACID, an actionconditional visual dynamics model for volumetric deformable objects based on structured implicit neural representations. ACID integrates two new techniques: implicit representations for action-conditional dynamics and geodesics-based contrastive learning. To represent deformable dynamics from partial RGB-D observations, we learn implicit representations of occupancy and flow-based forward dynamics. To accurately identify state change under large non-rigid deformations, we learn a correspondence embedding field through a novel geodesics-based contrastive loss. To evaluate our approach, we develop a simulation framework for manipulating complex deformable shapes in realistic scenes and a benchmark containing over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions over existing approaches. The ACID dynamics models are successfully employed to goal-conditioned deformable manipulation tasks, resulting in a 30% increase in task success rate over the strongest baseline. For more results and information, please visit https://b0ku1.github.io/acid/.

show abstract

Section: Discussion Of Limitationsmentioning

confidence: 99%

“…Alternative approaches try to tackle deformable objects manipulation with learned dynamics models in pixel [31]- [33] or latent [34]- [37] spaces, rather than explicit physical computation. Pixel-space models directly predict future observations and measure prediction quality with the reconstruction loss or the perceptual loss.…”

Section: Related Workmentioning

confidence: 99%

ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation

Shen¹,

Jiang²,

Choy³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Building on the visual foresight frameworks, [8] proposes the VisuoSpatial Foresight which integrates the depth map information with the pure RGB data to learn the visual dynamic model of fabrics in a simulated environment. An extension of this approach is given in [9] where the main states of the framework are improved.…”

Section: Related Workmentioning

confidence: 99%

Augment-Connect-Explore: a Paradigm for Visual Action Planning with Data Scarcity

Lippi¹,

Welle²,

Poklukar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual action planning particularly excels in applications where the state of the system cannot be computed explicitly, such as manipulation of deformable objects, as it enables planning directly from raw images. Even though the field has been significantly accelerated by deep learning techniques, a crucial requirement for their success is the availability of a large amount of data. In this work, we propose the Augment-Connect-Explore (ACE) paradigm to enable visual action planning in cases of data scarcity. We build upon the Latent Space Roadmap (LSR) framework which performs planning with a graph built in a low dimensional latent space. In particular, ACE is used to i) Augment the available training dataset by autonomously creating new pairs of datapoints, ii) create new unobserved Connections among representations of states in the latent graph, and iii) Explore new regions of the latent space in a targeted manner. We validate the proposed approach on both simulated box stacking and real-world folding task showing the applicability for rigid and deformable object manipulation tasks, respectively.

show abstract

“…However, due to deformable objects' high degrees of freedom (DoF) and consequent challenges in state estimation and dynamics modeling, manipulating deformable objects requires significant innovations beyond the typical robotic paradigm that focuses only on rigid objects. Recent advances show promising results in manipulating clothes [38,36,60,15,63,17] and ropes [62,57], yet the manipulation of objects with high plasticity, such as dough or plasticine, poses a unique set of challenges and is currently underexplored [37], despite the ubiquity of such objects in household and industrial settings. In this paper, we investigate how to empower robots to model and manipulate elasto-plastic objects based on raw RGBD visual observations.…”

Section: Introductionmentioning

confidence: 99%

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Shi¹,

Xu²,

Huang³

et al. 2022

Robotics: Science and Systems XVIII

View full text Add to dashboard Cite

Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned modelbased planning framework is comparable to and sometimes better than human subjects on the tested tasks. 1

show abstract

VisuoSpatial Foresight for physical sequential fabric manipulation

Cited by 30 publications

References 57 publications

ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation

ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation

Augment-Connect-Explore: a Paradigm for Visual Action Planning with Data Scarcity

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Contact Info

Product

Resources

About