Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making

Seita, Daniel; Jamali, Nawid; Laskey, Michael; Tanwani, Ajay Kumar; Berenstein, Ron; Baskaran, Prakash; Iba, Soshi; Canny, John; Goldberg, Ken

doi:10.48550/arxiv.1809.09810

Cited by 5 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work contributes a method to directly predict OBBs from a neural network. A representation that is used for deformable manipulation are keypoints or learned correspondences, which have seen significant success in tasks such as deformable manipulation [5,10,12,17,18,24] and grasping [25,26]. Ganapathi et al [10] and Sundaresan et al [5] predict correspondences from domain-randomized depth or monocular RGB images and demonstrate impressive transfer to real fabrics and ropes in constrained lab environments.…”

Section: Parameterized Representations For Manipulationmentioning

confidence: 99%

“…Note for the 9-DOF pose loses, we only enforce them where the Gaussian heatmaps are greater than 0.3, to prevent ambiguity in empty space. Keypoints: Keypoints and learned correspondences are a common representation for robot manipulation, especially in deformable manipulation [5,10,17,18,25]. SimNet has an output head that predicts keypoints of various classes, which can be fed into planners for manipulation.…”

Section: Extracting High-level Predictions For Manipulation Tasksmentioning

confidence: 99%

“…This task is challenging to perform using depth sensing, because the depth resolution of most commercial depth sensors cannot capture the subtle variations in depth due to the thickness of a t-shirt. Keypoints are a popular representation for manipulating deformable objects [17,18]. We parameterize a robot shirt folding policy using keypoint predictions for the shirt's neck, sleeves, and bottom corners (Sec.…”

Section: Grasp Point Prediction In T-shirt Foldingmentioning

confidence: 99%

“…The third component is a heuristic method for predicting grasp positions from the OBB and keypoint predictions. These grasp positions are combined with a classical planner to execute manipulation behaviors [5,10,17,18,19].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Kollar,

Laskey,

Stone

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Robot manipulation of unknown objects in unstructured environments is a challenging problem due to the variety of shapes, materials, arrangements and lighting conditions. Even with large-scale real-world data collection, robust perception and manipulation of transparent and reflective objects across various lighting conditions remains challenging. To address these challenges we propose an approach to performing sim-to-real transfer of robotic perception. The underlying model, SimNet, is trained as a single multi-headed neural network using simulated stereo data as input and simulated object segmentation masks, 3D oriented bounding boxes (OBBs), object keypoints and disparity as output. A key component of SimNet is the incorporation of a learned stereo sub-network that predicts disparity. SimNet is evaluated on 2D car detection, unknown object detection and deformable object keypoint detection and significantly outperforms a baseline that uses a structured light RGB-D sensor. By inferring grasp positions using the OBB and keypoint predictions, SimNet can be used to perform end-to-end manipulation of unknown objects in both "easy" and "hard" scenarios using our fleet of Toyota HSR robots in four home environments. In unknown object grasping experiments, the predictions from the baseline RGB-D network and SimNet enable successful grasps of most of the "easy" objects. However, the RGB-D baseline only grasps 35% of the "hard" (e.g., transparent) objects, while SimNet grasps 95%, suggesting that SimNet can enable robust manipulation of unknown objects, including transparent objects, in unknown environments. Additional visualizations and materials are located at https://tinyurl.com/simnet-corl.

show abstract

Section: Parameterized Representations For Manipulationmentioning

confidence: 99%

Section: Extracting High-level Predictions For Manipulation Tasksmentioning

confidence: 99%

Section: Grasp Point Prediction In T-shirt Foldingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Kollar,

Laskey,

Stone

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Deformable object manipulation has been a long standing problem [9]- [13], with two unique challenges. First, in contrast with rigid objects, there is no obvious representation of state.…”

Section: Introductionmentioning

confidence: 99%

Learning to Manipulate Deformable Objects without Demonstrations

Wu¹,

Yan²,

Kurutach³

et al. 2019

Preprint

View full text Add to dashboard Cite

In this paper we tackle the problem of deformable object manipulation through model-free visual reinforcement learning (RL). In order to circumvent the sample inefficiency of RL, we propose two key ideas that accelerate learning. First, we propose an iterative pick-place action space that encodes the conditional relationship between picking and placing on deformable objects. The explicit structural encoding enables faster learning under complex object dynamics. Second, instead of jointly learning both the pick and the place locations, we only explicitly learn the placing policy conditioned on random pick points. Then, by selecting the pick point that has Maximal Value under Placing (MVP), we obtain our picking policy. Using this learning framework, we obtain an order of magnitude faster learning compared to independent action-spaces on our suite of deformable object manipulation tasks. Finally, using domain randomization, we transfer our policies to a real PR2 robot for challenging cloth and rope manipulation.

show abstract

VIRDO: Visio-tactile Implicit Representations of Deformable Objects

Wi¹,

Florence²,

Zeng³

et al. 2022

2022 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief distribution of deformation within a trajectory, and (ii) enables practical real-world application by removing the need for contact patches. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.

show abstract

Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making

Cited by 5 publications

References 0 publications

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Learning to Manipulate Deformable Objects without Demonstrations

VIRDO: Visio-tactile Implicit Representations of Deformable Objects

Contact Info

Product

Resources

About