Sample-Efficient Learning of Deformable Linear Object Manipulation in the Real World Through Self-Supervision

Lee, Robert; Hamaya, Masashi; Murooka, Takayuki; Ijiri, Yoshihisa; Corke, Peter

doi:10.1109/lra.2021.3130377

Cited by 12 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most prior works use human demonstrations or robot explorations to train controlling policies for different tasks. [27], [39], and [17] fed human-made demonstrations to robots for learning control policies for shape control and knot-tying. Due to the tedium of constructing manual demonstrations, some researchers take advantage of the robots' automation to learn a policy purely from robotic exploration [49,53].…”

Section: Related Workmentioning

confidence: 99%

Sim2Real Physically Informed Neural Controllers for Robotic Deployment of Deformable Linear Objects

Tong¹,

Choi²,

Qin³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deformable linear objects, such as rods, cables, and ropes, play important roles in daily life.However, manipulation of DLOs is challenging as large geometrically nonlinear deformations may occur during the manipulation process. This problem is made even more difficult as the different deformation modes (e.g., stretching, bending, and twisting) may result in elastic instabilities during manipulation. In this paper, we formulate a physics-guided data-driven method to solve a challenging manipulation task -accurately deploying a DLO (an elastic rod) onto a rigid substrate along various prescribed patterns. Our framework combines machine learning, scaling analysis, and physics-based simulations to develop a physically informed neural controller for deployment.We explore the complex interplay between the gravitational and elastic energies of the manipulated DLO and obtain a control method for DLO deployment that is robust against friction and material properties. Out of the numerous geometrical and material properties of the rod and substrate, we show that only three non-dimensional parameters are needed to describe the deployment process with physical analysis. Therefore, the essence of the controlling law for the manipulation task can be constructed with a low-dimensional model, drastically increasing the computation speed. The effectiveness of our optimal control scheme is shown through a comprehensive robotic case study comparing against a heuristic control method for deploying rods for a wide variety of patterns.In addition to this, we also showcase the practicality of our control scheme by having a robot accomplish challenging high-level tasks such as mimicking human handwriting and tying knots.

show abstract

Section: Related Workmentioning

confidence: 99%

Sim2Real Physically Informed Neural Controllers for Robotic Deployment of Deformable Linear Objects

Tong¹,

Choi²,

Qin³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Deformable object manipulation is a long-standing challenge in robotics. However, many existing methods focus on objects such as ropes [59,45,61,57,24,3,61], cables [54,48,53], clothes [31,38,36,60,15,2,1], and gauze [58]. By contrast, we investigate the less-explored manipulation of elasto-plastic objects, such as plasticine, which are only studied in limited previous works [49,27].…”

Section: B Manipulating Deformable Objectsmentioning

confidence: 99%

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Shi¹,

Xu²,

Huang³

et al. 2022

Robotics: Science and Systems XVIII

View full text Add to dashboard Cite

Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned modelbased planning framework is comparable to and sometimes better than human subjects on the tested tasks. 1

show abstract

“…However, recent progress in cloth manipulation has been impressive due to advancements in learning-based approaches [12]- [15]. In particular, spatial pick-and-place action spaces allow for the use of Fully Convolutional Networks for learning action affordance heatmaps [1] as well as dynamics models [16] to achieve sample efficiency. Such architectures have also been used to learn fabric smoothing via flinging [17], bimanual stretching [18] and pick-and-place [2], as well as one-step fabric folding policies [19].…”

Section: A Deformable Object Manipulationmentioning

confidence: 99%

“…Several approaches have achieved impressive results for reaching goals with dynamics models [16], [20], [21]. [19] learn a flow-based model for predicting single timestep folds, making use of simulated cloth ground-truth positions.…”

Section: A Deformable Object Manipulationmentioning

confidence: 99%

“…The robot observes the object resting on a flat workspace from an overhead camera, and performs pick-and-place actions, corresponding directly to locations in the image. This is referred to as a spatial action space [25], and is widely used in deformable object manipulation [1], [2], [16], [19]. Our objective is to find the optimal pickand-place action for the task given an image observation.…”

Section: Approach a Problem Definitionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Fabric Manipulation in the Real World with Human Videos

Lee¹,

Abou-Chakra²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising alternative is to learn fabric manipulation directly from watching humans perform the task. In this work, we explore how demonstrations for fabric manipulation tasks can be collected directly by humans, providing an extremely natural and fast data collection pipeline. Then, using only a handful of such demonstrations, we show how a pick-and-place policy can be learned and deployed on a real robot, without any robot data collection at all. We demonstrate our approach on a fabric folding task, showing that our policy can reliably reach folded states from crumpled initial configurations. Videos are available at: https://sites.google.com/view/foldingbyhand

show abstract

Sample-Efficient Learning of Deformable Linear Object Manipulation in the Real World Through Self-Supervision

Cited by 12 publications

References 24 publications

Sim2Real Physically Informed Neural Controllers for Robotic Deployment of Deformable Linear Objects

Sim2Real Physically Informed Neural Controllers for Robotic Deployment of Deformable Linear Objects

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Learning Fabric Manipulation in the Real World with Human Videos

Contact Info

Product

Resources

About