Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

Sundaresan, Priya; Grannen, Jennifer; Thananjeyan, Brijen; Laskey, Michael; Stone, Kevin; Gonzalez, Joseph E.; Goldberg, Ken

doi:10.1109/icra40945.2020.9197121

Cited by 90 publications

(82 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A neural network model is trained to predict pushing points, given current and desired shape contours of the sand. Synthetic data has been used to learn collaborative manipulation of a cloth (42) and tying a rope (115). Reinforcement learning RL acquires a control policy through trial-and-error interactions, e.g., episodic rollouts.…”

Section: Imitation Learningmentioning

confidence: 99%

Modeling, learning, perception, and control methods for deformable object manipulation

2021

View full text Add to dashboard Cite

Perceiving and handling deformable objects is an integral part of everyday life for humans. Automating tasks such as food handling, garment sorting, or assistive dressing requires open problems of modeling, perceiving, planning, and control to be solved. Recent advances in data-driven approaches, together with classical control and planning, can provide viable solutions to these open challenges. In addition, with the development of better simulation environments, we can generate and study scenarios that allow for benchmarking of various approaches and gain better understanding of what theoretical developments need to be made and how practical systems can be implemented and evaluated to provide flexible, scalable, and robust solutions. To this end, we survey more than 100 relevant studies in this area and use it as the basis to discuss open problems. We adopt a learning perspective to unify the discussion over analytical and data-driven approaches, addressing how to use and integrate model priors and task data in perceiving and manipulating a variety of deformable objects.

show abstract

Section: Imitation Learningmentioning

confidence: 99%

Modeling, learning, perception, and control methods for deformable object manipulation

2021

View full text Add to dashboard Cite

show abstract

“…Deformable object manipulation has seen a surge of interest recently in the robotics research community [18,34,33,28,6,10,25,26], but modeling state and action spaces in such tasks remains challenging. Much prior work focuses on the complex task of generating plans for manipulating cables and higher-dimensional deformable objects.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, Lui and Saxena [19] and Chi and Berenson [3] propose using classical visual feature extraction to estimate the state of deformable rope and cloth, respectively, subject to partial occlusion. Sundaresan et al [28] investigate object representation learning via dense object descriptors [24,5] for rope knot-tying and arrangement, and Ganapathi et al [6] extend this methodology to 2D fabric smoothing and folding. Alternative perceptiondriven approaches explore learning latent state spaces [34] for cloth manipulation, or using semantic keypoint perception [7,33] for rope tracking.…”

Section: A Deformable Object Manipulationmentioning

confidence: 99%

“…As robots automate more tasks involving 1D deformable objects, which we refer to as "cables," they will increasingly confront highly complex knots, either because the task itself requires knot untangling or because untangling cables is a prerequisite for a downstream task. Prior work has explored untangling manipulation plans consisting of highlevel manipulation primitives (e.g., pick and pull points) [28,19,7], but robots often face challenges when attempting to execute these plans in physical manipulation tasks, due to the complexity of dynamics modeling and perception in real-world settings. This motivates the need for a low-level controller which can mitigate the gap between high-level planning and physical execution.…”

Section: Introductionmentioning

confidence: 99%

“…These properties complicate the problems of both generating untangling manipulation plans and executing these plans effectively, especially in densely knotted configurations. Prior work for cable manipulation has primarily explored manipulation planning methods that learn full state information [28,19] or task-specific features [7]. These methods then execute planned actions through a sequence of analytical, hand-engineered motion primitives, which can be brittle and imprecise under increasing knot complexity and density due to layered self-occlusions, friction at cable intersections, and the need for precision in grasping.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies

Sundaresan

Grannen

Thananjeyan

et al. 2021

Robotics: Science and Systems XVII

Self Cite

View full text Add to dashboard Cite

Robot manipulation for untangling 1D deformable structures such as ropes, cables, and wires is challenging due to their infinite dimensional configuration space, complex dynamics, and tendency to self-occlude. Analytical controllers often fail in the presence of dense configurations, due to the difficulty of grasping between adjacent cable segments. We present two algorithms that enhance robust cable untangling, LOKI and SPi-DERMan, which operate alongside HULK, a high-level planner from prior work. LOKI uses a learned model of manipulation features to refine a coarse grasp keypoint prediction to a precise, optimized location and orientation, while SPiDERMan uses a learned model to sense task progress and apply recovery actions. We evaluate these algorithms in physical cable untangling experiments with 336 knots and over 1500 actions on real cables using the da Vinci surgical robot. We find that the combination of HULK, LOKI, and SPiDERMan is able to untangle dense overhand, figure-eight, double-overhand, square, bowline, granny, stevedore, and triple-overhand knots. The composition of these methods successfully untangles a cable from a dense initial configuration in 68.3% of 60 physical experiments and achieves 50% higher success rates than baselines from prior work. Supplementary material, code, and videos can be found at https://tinyurl.com/rssuntangling.

show abstract

Task attention-based multimodal fusion and curriculum residual learning for context generalization in robotic assembly

Wang,

Lin,

Liu

et al. 2024

Appl Intell

View full text Add to dashboard Cite

Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

Cited by 90 publications

References 33 publications

Modeling, learning, perception, and control methods for deformable object manipulation

Modeling, learning, perception, and control methods for deformable object manipulation

Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies

Task attention-based multimodal fusion and curriculum residual learning for context generalization in robotic assembly

Contact Info

Product

Resources

About