Learning Affordance Segmentation for Real-World Robotic Manipulation via Synthetic Images

Chu, Fu-Jen; Xu, Ruinian; Vela, Patricio A.

doi:10.1109/lra.2019.2894439

Cited by 43 publications

(14 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With recent advances in deep learning, many efforts resort to semantic part segmentation [6], [7] or keypoint detection [8] via supervised learning on manually labeled real-world data. A line of research is circumventing the complicated data collection process by training in simulation [9], [10].…”

Section: Novel Unseen Objectsmentioning

confidence: 99%

“…This, however, often assumes manually annotated real world data is available to perform supervised training [33], [34], which is costly and time-consuming to obtain. While [6], [35] alleviates the problem via sim-to-real transfer, it still requires manual specification of semantic parts on 3D models for generating synthetic affordance labels. Instead, another line of research [9], [10], [36] proposed to learn semantic tool manipulation via self-interaction in simulation.…”

Section: Novel Unseen Objectsmentioning

confidence: 99%

See 1 more Smart Citation

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Wen¹,

Lian²,

Bekris³

et al. 2021

Preprint

View full text Add to dashboard Cite

Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data will be released upon acceptance at https://sites.google.com/view/catgrasp.

show abstract

Section: Novel Unseen Objectsmentioning

confidence: 99%

Section: Novel Unseen Objectsmentioning

confidence: 99%

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Wen¹,

Lian²,

Bekris³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Stoytchev [15] introduces an approach to ground tool affordances via dynamically applying different behaviors from a behavioral repertoire. [9], [10], [14], [16], [17] use convolutional neural networks (CNNs) to detect regions of affordance in an image. Ruiz and Mayol-Cuevas [11] predict affordance candidate locations in environments via the interaction tensor.…”

Section: Related Workmentioning

confidence: 99%

Put the Bear on the Chair! Intelligent Robot Interaction with Previously Unseen Chairs via Robot Imagination

Wu¹,

Meng²,

Ruan³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this letter, we study the problem of autonomously placing a teddy bear on a previously unseen chair for sitting. To achieve this goal, we present a novel method for robots to imagine the sitting pose of the bear by physically simulating a virtual humanoid agent sitting on the chair. We also develop a robotic system which leverages motion planning to plan SE(2) motions for a humanoid robot to walk to the chair and whole-body motions to put the bear on it, respectively. Furthermore, to cope with the cases where the chair is not in an accessible pose for placing the bear, a human-robot interaction (HRI) framework is introduced in which a human follows language instructions given by the robot to rotate the chair and help make the chair accessible. We implement our method with a robot arm and a humanoid robot. We calibrate the proposed system with 3 chairs and test on 12 previously unseen chairs in both accessible and inaccessible poses extensively. Results show that our method enables the robot to autonomously put the teddy bear on the 12 unseen chairs with a very high success rate. The HRI framework is also shown to be very effective in changing the accessibility of the chair. Source code will be available. Video demos are available at https: //chirikjianlab.github.io/putbearonchair/.

show abstract

“…Antanas et al [13] encode suitable grasp regions based on probabilistic logic descriptions of tasks and segmented object parts. Works on object affordance detection [17], [18], [19], [20], [21] have leveraged the affordances of object parts to define the correspondences between affordances and grasp types (e.g., rim grasp for parts with contain or scoop affordance.) Detry et al [22] built on these works by training a separate detection model using data generated by predefined heuristics to detect suitable grasp regions for each task.…”

Section: A Semantic Graspingmentioning

confidence: 99%

“…Sawatzky et al [25] tried to reduce the cost of labeling with weakly supervised learning. Chu et al [18] explored transferring from systhetic data to real environment with unsupervised domain adaptation. We are the first to leverage part affordances to learn semantic grasping from data.…”

Section: B Affordance Detectionmentioning

confidence: 99%

CAGE: Context-Aware Grasping Engine

Liu

Daruna

Chernova

2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Semantic grasping is the problem of selecting stable grasps that are functionally suitable for specific object manipulation tasks. In order for robots to effectively perform object manipulation, a broad sense of context, including object and task constraints, needs to be accounted for. We introduce the Context-Aware Grasping Engine, a neural network structure based on the Wide & Deep model, to learn suitable semantic grasps from data. We quantitatively validate our approach against three prior methods on a novel dataset consisting of 14,000 semantic grasps for 44 objects, 7 tasks, and 6 different object states. Our approach outperformed all baselines by statistically significant margins, producing new insights into the importance of balancing memorization and generalization of contexts for semantic grasping. We further demonstrate the effectiveness of our approach on robot experiments in which the presented model successfully achieved 31 of 32 grasps.

show abstract

Learning Affordance Segmentation for Real-World Robotic Manipulation via Synthetic Images

Cited by 43 publications

References 35 publications

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Put the Bear on the Chair! Intelligent Robot Interaction with Previously Unseen Chairs via Robot Imagination

CAGE: Context-Aware Grasping Engine

Contact Info

Product

Resources

About