Learning where to trust unreliable models in an unstructured world for deformable object manipulation

Mitrano, Peter; McConachie, Dale; Berenson, Dmitry

doi:10.1126/scirobotics.abd8170

Cited by 38 publications

(23 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3) Cluttered environment: To the author's knowledge, few model-based approaches have been proposed in the literature due to complexity of this problem. Mitrano et al [26] proposed to train a classifier to determine where the offline learned model is in effect and utilize a learned policy to recover the robot when the model is not reliable. However, this approach cannot deal with the scenario that the cable requires to contact with the environment.…”

Section: Experiments Resultsmentioning

confidence: 99%

Offline-Online Learning of Deformation Model for Cable Manipulation With Graph Neural Networks

Wang

Zhang

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Manipulating deformable linear objects by robots has a wide range of applications, e.g., manufacturing and medical surgery. To complete such tasks, an accurate dynamics model for predicting the deformation is critical for robust control. In this work, we deal with this challenge by proposing a hybrid offline-online method to learn the dynamics of cables in a robust and data-efficient manner. In the offline phase, we adopt Graph Neural Network (GNN) to learn the deformation dynamics purely from the simulation data. Then a linear residual model is learned in real-time to bridge the sim-to-real gap. The learned model is then utilized as the dynamics constraint of a trust region based Model Predictive Controller (MPC) to calculate the optimal robot movements. The online learning and MPC run in a closed-loop manner to robustly accomplish the task. Finally, comparative results with existing methods are provided to quantitatively show the effectiveness and robustness. Experiment videos are available at https://msc.berkeley.edu/research/deformable-GNN.html.

show abstract

Section: Experiments Resultsmentioning

confidence: 99%

Offline-Online Learning of Deformation Model for Cable Manipulation With Graph Neural Networks

Wang

Zhang

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…Since the purpose of our augmentations is to improve performance using small datasets, it is important that this number is small. In contrast, prior work learning a similar classifier used over 100,000 examples in their datasets [13,12]. This is similar to the number of training examples we have after augmentation, which is 75,950 on average.…”

Section: B Bimanual Rope Manipulationmentioning

confidence: 93%

“…In this task, the end points of a rope are held by the robot in its grippers in a scene resembling the engine bay of a car, similar to [13], and shown in Figure 4. The robot has two 7-dof arms attached to a 3-dof torso with parallel-jaw grippers.…”

Section: B Bimanual Rope Manipulationmentioning

confidence: 99%

See 1 more Smart Citation

Data Augmentation for Manipulation

Mitrano¹,

Berenson²

2022

Preprint

View full text Add to dashboard Cite

The success of deep learning depends heavily on the availability of large datasets, but in robotic manipulation there are many learning problems for which such datasets do not exist. Collecting these datasets is time-consuming and expensive, and therefore learning from small datasets is an important open problem. Within computer vision, a common approach to a lack of data is data augmentation. Data augmentation is the process of creating additional training examples by modifying existing ones. However, because the types of tasks and data differ, the methods used in computer vision cannot be easily adapted to manipulation. Therefore, we propose a data augmentation method for robotic manipulation. We argue that augmentations should be valid, relevant, and diverse. We use these principles to formalize augmentation as an optimization problem, with the objective function derived from physics and knowledge of the manipulation domain. This method applies rigid body transformations to trajectories of geometric state and action data. We test our method in two scenarios: 1) learning the dynamics of planar pushing of rigid cylinders, and 2) learning a constraint checker for rope manipulation. These two scenarios have different data and label types, yet in both scenarios, training on our augmented data significantly improves performance on downstream tasks. We also show how our augmentation method can be used on real-robot data to enable more data-efficient online learning.

show abstract

“…We demonstrate the proposed method is successful in filtering out distracting data and that the resulting trained model is more accurate in the regions of state-action space where the source and target dynamics are similar. The second contribution is a data-efficient online-learning method that pairs our adaptation method with prior work on planning with unreliable dynamics models [6], [7]. We call our combined method for online learning FOCUS.…”

Section: Introductionmentioning

confidence: 99%

Focused Adaptation of Dynamics Models for Deformable Object Manipulation

Mitrano¹,

Alex²,

Kroemer³

et al. 2022

Preprint

View full text Add to dashboard Cite

In order to efficiently learn a dynamics model for a task in a new environment, one can adapt a model learned in a similar source environment. However, existing adaptation methods can fail when the target dataset contains transitions where the dynamics are very different from the source environment. For example, the source environment dynamics could be of a rope manipulated in free-space, whereas the target dynamics could involve collisions and deformation on obstacles. Our key insight is to improve data efficiency by focusing model adaptation on only the regions where the source and target dynamics are similar. In the rope example, adapting the free-space dynamics requires significantly fewer data than adapting the free-space dynamics while also learning collision dynamics. We propose a new method for adaptation that is effective in adapting to regions of similar dynamics. Additionally, we combine this adaptation method with prior work on planning with unreliable dynamics to make a method for data-efficient online adaptation, called FOCUS. We first demonstrate that the proposed adaptation method achieves statistically significantly lower prediction error in regions of similar dynamics on simulated rope manipulation and plant watering tasks. We then show on a bimanual rope manipulation task that FOCUS achieves data-efficient online learning, in simulation and in the real world.

show abstract

Learning where to trust unreliable models in an unstructured world for deformable object manipulation

Cited by 38 publications

References 35 publications

Offline-Online Learning of Deformation Model for Cable Manipulation With Graph Neural Networks

Offline-Online Learning of Deformation Model for Cable Manipulation With Graph Neural Networks

Data Augmentation for Manipulation

Focused Adaptation of Dynamics Models for Deformable Object Manipulation

Contact Info

Product

Resources

About