Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making

Seita, Daniel; Jamali, Nawid; Laskey, Michael; Tanwani, Ajay Kumar; Berenstein, Ron; Baskaran, Prakash; Iba, Soshi; Canny, John; Goldberg, Ken

doi:10.1007/978-3-030-95459-8_17

Cited by 58 publications

(52 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, other approaches circumvent state estimation by performing end-to-end visuomotor learning for goal-conditioned tasks. These include rope shape-matching and knot-tying by learning dynamics models [16,17]; cable vaulting by behavioral cloning [27]; cloth smoothing and folding from video prediction models [3,8], latent dynamics models [12], reinforcement learning [11], and imitation learning [20,21]; and bag manipulation by inferring spatial displacements [19]. While general, these algorithms do not leverage the geometric structure specific to the cable manipulation problem, which makes them difficult to apply to highly complex tasks such as cable disentangling, in which finegrained perception and manipulation is critical for success.…”

Section: A Deformable Object Manipulationmentioning

confidence: 99%

Disentangling Dense Multi-Cable Knots

Viswanath¹,

Grannen²,

Sundaresan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Disentangling two or more cables requires many steps to remove crossings between and within cables. We formalize the problem of disentangling multiple cables and present an algorithm, Iterative Reduction Of Non-planar Multiple cAble kNots (IRON-MAN), that outputs robot actions to remove crossings from multi-cable knotted structures. We instantiate this algorithm with a learned perception system, inspired by prior work in single-cable untying that given an image input, can disentangle two-cable twists, three-cable braids, and knots of two or three cables, such as overhand, square, carrick bend, sheet bend, crown, and fisherman's knots. IRON-MAN keeps track of task-relevant keypoints corresponding to target cable endpoints and crossings and iteratively disentangles the cables by identifying and undoing crossings that are critical to knot structure. Using a da Vinci surgical robot, we experimentally evaluate the effectiveness of IRON-MAN on untangling multicable knots of types that appear in the training data, as well as generalizing to novel classes of multi-cable knots. Results suggest that IRON-MAN is effective in disentangling knots involving up to three cables with 80.5% success and generalizing to knot types that are not present during training, with cables of both distinct or identical colors.

show abstract

Section: A Deformable Object Manipulationmentioning

confidence: 99%

Disentangling Dense Multi-Cable Knots

Viswanath¹,

Grannen²,

Sundaresan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In contrast to prior work, we demonstrate that similar descriptors can be learned and leveraged for manipulation of very deformable 1D structures such as rope. We also learn descriptors from While [11,16,34] learn descriptors using color image input, we use synthetic depth input, which facilitates sim-to-real transfer of the learned representations [22,37] and richly encodes the geometric structure of ropes in knotted configurations. Fig.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Manipulating deformable objects is valuable for a wide variety of applications from surgery and manufacturing to household robotics [2,8,11,14,15,19,26,[37][38][39]. We specifically consider manipulation of rope, whose infinite dimensional configuration space objects makes it difficult to build accurate dynamical models.…”

Section: Introductionmentioning

confidence: 99%

Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

Sundaresan¹,

Grannen²,

Thananjeyan³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Robotic manipulation of deformable 1D objects such as ropes, cables, and hoses is challenging due to the lack of high-fidelity analytic models and large configuration spaces. Furthermore, learning end-to-end manipulation policies directly from images and physical interaction requires significant time on a robot and can fail to generalize across tasks. We address these challenges using interpretable deep visual representations for rope, extending recent work on dense object descriptors for robot manipulation. This facilitates the design of interpretable and transferable geometric policies built on top of the learned representations, decoupling visual reasoning and control. We present an approach that learns point-pair correspondences between initial and goal rope configurations, which implicitly encodes geometric structure, entirely in simulation from synthetic depth images. We demonstrate that the learned representation -dense depth object descriptors (DDODs)can be used to manipulate a real rope into a variety of different arrangements either by learning from demonstrations or using interpretable geometric policies. In 50 trials of a knot-tying task with the ABB YuMi Robot, the system achieves a 66% knot-tying success rate from previously unseen configurations. See https://tinyurl.com/rope-learning for supplementary material and videos.

show abstract

“…Even seemingly rigid objects like metal wires significantly deform during everyday interactions. As a result, there has been a growing interest in algorithms that can tackle deformable object manipulation [54,15,42,43,45,58,46,29,50].…”

Section: Introductionmentioning

confidence: 99%

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Yan,

Vangipuram,

Abbeel

et al. 2020

Preprint

View full text Add to dashboard Cite

Fig. 1: Example trajectories using our contrastive forward model for rope and cloth manipulation. The top two rows show rope manipulation from different start states to different goal states, while the bottom two rows show cloth manipulation using different colored cloths. Note that in the last row, the robot is manipulating a white cloth, but are method is able to still use a blue cloth as the goal image.

show abstract

Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making

Cited by 58 publications

References 25 publications

Disentangling Dense Multi-Cable Knots

Disentangling Dense Multi-Cable Knots

Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Contact Info

Product

Resources

About