An Affordance Keypoint Detection Network for Robot Manipulation

Xu, Ruinian; Chu, Fu-Jen; Tang, Chao; Liu, Weiyu; Vela, Patricio A.

doi:10.1109/lra.2021.3062560

Cited by 30 publications

(26 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Likewise, identifying a means to translate the 2D grasp representation to a full 3D grasp pose would remove the need for a top-down grasp and permit richer manipulation from more varied viewpoints. Recent work on affordances and keypoints (Xu et al, 2021) indicates that keypoints should work well for recovering SE(3) grasp frames. Lastly, introducing grasp quality neural networks (Mahler et al, 2017; Morrison et al, 2019) would further resolve the what question for cluttered scenarios.…”

Section: Discussionmentioning

confidence: 99%

GKNet: Grasp keypoint network for grasp candidates detection

Chu

Vela

2022

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

Contemporary grasp detection approaches employ deep learning to achieve robustness to sensor and object model uncertainty. The two dominant approaches design either grasp-quality scoring or anchor-based grasp recognition networks. This paper presents a different approach to grasp detection by treating it as keypoint detection in image-space. The deep network detects each grasp candidate as a pair of keypoints, convertible to the grasp representation g = { x, y, w, θ} T, rather than a triplet or quartet of corner points. Decreasing the detection difficulty by grouping keypoints into pairs boosts performance. To promote capturing dependencies between keypoints, a non-local module is incorporated into the network design. A final filtering strategy based on discrete and continuous orientation prediction removes false correspondences and further improves grasp detection performance. GKNet, the approach presented here, achieves a good balance between accuracy and speed on the Cornell and the abridged Jacquard datasets (96.9% and 98.39% at 41.67 and 23.26 fps). Follow-up experiments on a manipulator evaluate GKNet using four types of grasping experiments reflecting different nuisance sources: static grasping, dynamic grasping, grasping at varied camera angles, and bin picking. GKNet outperforms reference baselines in static and dynamic grasping experiments while showing robustness to varied camera viewpoints and moderate clutter. The results confirm the hypothesis that grasp keypoints are an effective output representation for deep grasp networks that provide robustness to expected nuisance factors.

show abstract

Section: Discussionmentioning

confidence: 99%

GKNet: Grasp keypoint network for grasp candidates detection

Chu

Vela

2022

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…That improves the relationship between grasping and objects though it cannot generate diverse grasp candidates automatically. On the basis of traditional pixel-wise part segmentation, [9] introduced an extra keypoint detection module, whose predictions consists of position, direction, and extent, guiding a more stable grasp pose. However, the problem of these pixel-based part affordance methods is that 6-DoF grasp detection is hard to embed in it, causing generated robotic grasp candidates in a very restricted workspace.…”

Section: Related Work Deep Visualmentioning

confidence: 99%

“…Rather than using different object parts as different affordance representations [9], [14], [16] , we annotate all successful grasps of selected objects with different affordance labels. For example, grasp affordance in the mug instance means all successful grasps around the mug handle while pour affordance means all successful grasps around the upper mug rim.…”

Section: A Affordance Grasp Dataset Constructionmentioning

confidence: 99%

Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance

Chen¹,

Liang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Currently, task-oriented grasp detection approaches are mostly based on pixel-level affordance detection and semantic segmentation. These pixel-level approaches heavily rely on the accuracy of a 2D affordance mask, and the generated grasp candidates are restricted to a small workspace. To mitigate these limitations, we firstly construct a novel affordance-based grasp dataset and propose a 6-DoF taskoriented grasp detection framework, which takes the observed object point cloud as input and predicts diverse 6-DoF grasp poses for different tasks. Specifically, our implicit estimation network and visual affordance network in this framework could directly predict coarse grasp candidates, and corresponding 3D affordance heatmap for each potential task, respectively. Furthermore, the grasping scores from coarse grasps are combined with heatmap values to generate more accurate and finer candidates. Our proposed framework shows significant improvements compared to baselines for existing and novel objects on our simulation dataset. Although our framework is trained based on the simulated objects and environment, the final generated grasp candidates can be accurately and stably executed in the real robot experiments when the object is randomly placed on a support surface.

show abstract

“…Task-Relevant Grasping -Task-relevant grasping requires the grasps to be compatible with downstream manipulation tasks. Prior works have developed frameworks to predict affordance segmentation [7], [27]- [31] or keypoints [32] over the observed image or point cloud. This, however, often assumes manually annotated real world data is available to perform supervised training [33], [34], which is costly and time-consuming to obtain.…”

Section: Novel Unseen Objectsmentioning

confidence: 99%

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Wen¹,

Lian²,

Bekris³

et al. 2021

Preprint

View full text Add to dashboard Cite

Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data will be released upon acceptance at https://sites.google.com/view/catgrasp.

show abstract

An Affordance Keypoint Detection Network for Robot Manipulation

Cited by 30 publications

References 35 publications

GKNet: Grasp keypoint network for grasp candidates detection

GKNet: Grasp keypoint network for grasp candidates detection

Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Contact Info

Product

Resources

About