Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

Ganapathi, Aditya; Sundaresan, Priya; Thananjeyan, Brijen; Balakrishna, Ashwin; Seita, Daniel; Grannen, Jennifer; Hwang, Mina; Hoque, Ryan; Jamali, Nawid; Yamane, Katsu; Iba, Soshi; Goldberg, Ken

doi:10.48550/arxiv.2003.12698

Cited by 5 publications

(13 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior works have proposed various hand-defined representations for cloth manipulation, such as parameterized shape models [28] or binary occupancy features [29]. Recent approaches use contrastive learning to learn pixel-wise latent embeddings for cloth [11,30]. Both contrastive learning [11] and goal-conditioned transporter networks [8] have been applied to imitate expert demonstrations.…”

Section: Related Workmentioning

confidence: 99%

“…Recent approaches use contrastive learning to learn pixel-wise latent embeddings for cloth [11,30]. Both contrastive learning [11] and goal-conditioned transporter networks [8] have been applied to imitate expert demonstrations. Our approach doesn't require expert actions, just sub-goal states provided at test-time to define the task.…”

Section: Related Workmentioning

confidence: 99%

“…3.1, and is similar to the protocol in Nair et al [7]. Our goals include test goals from Ganapathi et al [11] and Lee et al [9] that are achievable with one arm, as well as additional goals more suitable for two-arm actions (see Appendix Fig. S2 for the full set of goals).…”

Section: Simulation Experimentsmentioning

confidence: 99%

“…• A novel flow-based approach for learning goal-conditioned cloth manipulation policies that can perform dual-arm and single-arm actions. • A test suite for benchmarking goal-conditioned cloth folding algorithms encompassing and expanding on goals used in previous literature [4,9,11]; we perform extensive experiments using this test suite to evaluate FabricFlowNet (FFN), baselines [4,9], and ablations, demonstrating that FFN outperforms previous approaches. • Experiments to demonstrate that FFN generalizes to other cloth colors and shapes, even without training on such variations.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

Weng¹,

Bajracharya²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

We address the problem of goal-directed cloth manipulation, a challenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation in video, can also provide an effective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and as an action representation to improve performance. FabricFlowNet also elegantly switches between dual-arm and singlearm actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-art model-free and model-based cloth manipulation policies. We also present real-world experiments on a bimanual system, demonstrating effective sim-to-real transfer. Finally, we show that our method generalizes when trained on a single square cloth to other cloth shapes, such as T-shirts and rectangular cloths. Video and other supplementary materials are available at: https://sites.google.com/view/fabricflownet.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Simulation Experimentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

Weng¹,

Bajracharya²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To perform well in these settings, large-scale real-world data collection is often performed [1,2,3], which is costly, time-consuming, has limited flexibility and sometimes contains missing data (as is the case with RGB-D sensing of transparent objects). An alternative is to use simulation to automatically label a large dataset of images [2,4,5,6,7,8,9,10,11,12]. Although physics-based and photorealistic simulations can enable end-to-end learning methods, rendering can take significant computational resources and artists must generate high-quality artefacts [2,13].…”

Section: Introductionmentioning

confidence: 99%

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Kollar,

Laskey,

Stone

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Robot manipulation of unknown objects in unstructured environments is a challenging problem due to the variety of shapes, materials, arrangements and lighting conditions. Even with large-scale real-world data collection, robust perception and manipulation of transparent and reflective objects across various lighting conditions remains challenging. To address these challenges we propose an approach to performing sim-to-real transfer of robotic perception. The underlying model, SimNet, is trained as a single multi-headed neural network using simulated stereo data as input and simulated object segmentation masks, 3D oriented bounding boxes (OBBs), object keypoints and disparity as output. A key component of SimNet is the incorporation of a learned stereo sub-network that predicts disparity. SimNet is evaluated on 2D car detection, unknown object detection and deformable object keypoint detection and significantly outperforms a baseline that uses a structured light RGB-D sensor. By inferring grasp positions using the OBB and keypoint predictions, SimNet can be used to perform end-to-end manipulation of unknown objects in both "easy" and "hard" scenarios using our fleet of Toyota HSR robots in four home environments. In unknown object grasping experiments, the predictions from the baseline RGB-D network and SimNet enable successful grasps of most of the "easy" objects. However, the RGB-D baseline only grasps 35% of the "hard" (e.g., transparent) objects, while SimNet grasps 95%, suggesting that SimNet can enable robust manipulation of unknown objects, including transparent objects, in unknown environments. Additional visualizations and materials are located at https://tinyurl.com/simnet-corl.

show abstract

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks

Seita¹,

Florence²,

Tompson³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Rearranging and manipulating deformable objects such as cables, fabrics, and bags is a long-standing challenge in robotic manipulation. The complex dynamics and highdimensional configuration spaces of deformables, compared to rigid objects, make manipulation difficult not only for multistep planning, but even for goal specification. Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag". In this work, we develop a suite of simulated benchmarks with 1D, 2D, and 3D deformable structures, including tasks that involve image-based goal-conditioning and multi-step deformable manipulation. We propose embedding goal-conditioning into Transporter Networks, a recently proposed model architecture for learning robotic manipulation that rearranges deep features to infer displacements that can represent pick and place actions. We demonstrate that goal-conditioned Transporter Networks enable agents to manipulate deformable structures into flexibly specified configurations without test-time visual anchors for target locations. We also significantly extend prior results using Transporter Networks for manipulating deformable objects by testing on tasks with 2D and 3D deformables. Supplementary material is available at https://berkeleyautomation. github.io/bags/.

show abstract

Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

Cited by 5 publications

References 35 publications

FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks

Contact Info

Product

Resources

About