Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects

Ma, Xiao; Hsu, David; Lee, Wee Sun

doi:10.1109/icra46639.2022.9811597

Cited by 23 publications

(20 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These are used in tasks such as knotting [17,35] or untangling [12,57]. Manipulation of 2D objects refers to items such as clothing and fabrics, as studied in recent work on fabric smoothing [4,15,18,27,29,40,48,59,61], which often measuring quality using coverage. A smooth fabric with high coverage may make it easier to later do folding, another canonical task explored in prior work [1,10,24,31].…”

Section: A Deformable Object Manipulationmentioning

confidence: 99%

AutoBag: Learning to Open Plastic Bags and Insert Objects

Yunliang¹,

Baiyu²,

Seita³

et al. 2022

Preprint

View full text Add to dashboard Cite

Thin plastic bags are ubiquitous in retail stores, healthcare, food handling, recycling, homes, and school lunchrooms. They are challenging both for perception (due to specularities and occlusions) and for manipulation (due to the dynamics of their 3D deformable structure). We formulate the task of manipulating common plastic shopping bags with two handles from an unstructured initial state to a state where solid objects can be inserted into the bag for transport. We propose a self-supervised learning framework where a dual-arm robot learns to recognize the handles and rim of plastic bags using UV-fluorescent markings; at execution time, the robot does not use UV markings or UV light. We propose Autonomous Bagging (AutoBag), where the robot uses the learned perception model to open plastic bags through iterative manipulation. We present novel metrics to evaluate the quality of a bag state and new motion primitives for reorienting and opening bags from visual observations. In physical experiments, a YuMi robot using AutoBag is able to open bags and achieve a success rate of 16/30 for inserting at least one item across a variety of initial bag configurations. Supplementary material is available at https://sites.google.com/view/autobag.

show abstract

Section: A Deformable Object Manipulationmentioning

confidence: 99%

AutoBag: Learning to Open Plastic Bags and Insert Objects

Yunliang¹,

Baiyu²,

Seita³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With the rise of deep learning, researchers have recently employed data driven techniques to obtain large amounts of interaction data with cloth to learn manipulation policies using powerful function approximators, often with the help of simulators [27,47]. These works tend to learn quasistatic pick-and-place policies, which allow the cloth to settle between robot actions [13,17,26,29,36,42,43,[51][52][53]. Other researchers have learned continuous servoing policies [33], dynamic policies [15] or have explored learning cloth manipulation from purely real world interaction [25].…”

Section: A Cloth Manipulation Policiesmentioning

confidence: 99%

Learning to Singulate Layers of Cloth using Tactile Feedback

Tirumala¹,

Weng²,

Seita³

et al. 2022

Preprint

View full text Add to dashboard Cite

Robotic manipulation of cloth has applications ranging from fabrics manufacturing to handling blankets and laundry. Cloth manipulation is challenging for robots largely due to their high degrees of freedom, complex dynamics, and severe self-occlusions when in folded or crumpled configurations. Prior work on robotic manipulation of cloth relies primarily on vision sensors alone, which may pose challenges for fine-grained manipulation tasks such as grasping a desired number of cloth layers from a stack of cloth. In this paper, we propose to use tactile sensing for cloth manipulation; we attach a tactile sensor (ReSkin) to one of the two fingertips of a Franka robot and train a classifier to determine whether the robot is grasping a specific number of cloth layers. During test-time experiments, the robot uses this classifier as part of its policy to grasp one or two cloth layers using tactile feedback to determine suitable grasping points. Experimental results over 180 physical trials suggest that the proposed method outperforms baselines that do not use tactile feedback and has better generalization to unseen cloth compared to methods that use image classifiers. Code, data, and videos are available at https://sites.google.com/view/reskin-cloth.

show abstract

“…Similarly, Qi et al [37] use an autoencoder to learn a compact representation of DLOs shape which is used in a vision-based controller for manipulation. Ma et al [38] propose latent graph dynamics for deformable object manipulation which abstracts the deformable object state as a low-dimensional keypointbased graph with learned latent features. Learning latent state shows promising results in manipulation, however, it has not been applied to the DLO tracking problem.…”

Section: Dlo Latent Representationsmentioning

confidence: 99%

Particle Filters in Latent Space for Robust Deformable Linear Object Tracking

Yang

Stork

Stoyanov

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Tracking of deformable linear objects (DLOs) is important for many robotic applications. However, achieving robust and accurate tracking is challenging due to the lack of distinctive features or appearance on the DLO, the object's high-dimensional state space, and the presence of occlusion. In this letter, we propose a method for tracking the state of a DLO by applying a particle filter approach within a lower-dimensional state embedding learned by an autoencoder. The dimensionality reduction preserves state variation, while simultaneously enabling a particle filter to accurately track DLO state evolution with a practically feasible number of particles. Compared to previous works, our method requires neither running a high-fidelity physics simulation, nor manual designs of constraints and regularization. Without the assumption of knowing the initial DLO state, our method can achieve accurate tracking even under complex DLO motions and in the presence of severe occlusions. Our implementation is available at https://amm.aass.oru.se/dlo-pf-tracking/.

show abstract

Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects

Cited by 23 publications

References 15 publications

AutoBag: Learning to Open Plastic Bags and Insert Objects

AutoBag: Learning to Open Plastic Bags and Insert Objects

Learning to Singulate Layers of Cloth using Tactile Feedback

Particle Filters in Latent Space for Robust Deformable Linear Object Tracking

Contact Info

Product

Resources

About