Learning Grasp Affordance Reasoning Through Semantic Relations

Abstract

Reasoning about object affordances allows an autonomous agent to perform generalised manipulation tasks among object instances. While current approaches to grasp affordance estimation are effective, they are limited to a single hypothesis. We present an approach for detection and extraction of multiple grasp affordances on an object via visual input. We define semantics as a combination of multiple attributes, which yields benefits in terms of generalisation for grasp affordance prediction. We use Markov Logic Networks to build a knowledge base graph representation to obtain a probability distribution of grasp affordances for an object. To harvest the knowledge base, we collect and make available a novel dataset that relates different semantic attributes. We achieve reliable mappings of the predicted grasp affordances on the object by learning prototypical grasping patches from several examples. We show our method's generalisation capabilities on grasp affordance prediction for novel instances and compare with similar methods in the literature. Moreover, using a robotic platform, on simulated and real scenarios, we evaluate the success of the grasping task when conditioned on the grasp affordance prediction.

Robotics Dexterous Grasping: The Methods Based on Point Cloud and Deep Learning

“…Li Y. et al ( 2020 ) proposes a Deep Residual U-Nets on the basis of residual modules (He et al, 2016 ) to predict the graspable region of object, which is followed by a K-means (Lloyd, 1982 ) model clusters the graspable point cloud and the center of each cluster is leveraged as a grasp point. Ardón et al ( 2019 ) employs Markov logic networks (MLN) (Richardson and Domingos, 2006 ) for knowing the relationship between diverse objects and a pre-trained Res-Net (He et al, 2016 ) is utilized to accomplish object perception and feature extraction for querying grasp affordances by Gibbs sampling (Kim and Nelson, 1999 ). The main thought back Ardón et al ( 2019 ) is sampling several grasp affordances and evaluate them, the affordance with highest possibility will be selected and corresponding grasp configuration is calculated.…”

“…Ardón et al ( 2019 ) employs Markov logic networks (MLN) (Richardson and Domingos, 2006 ) for knowing the relationship between diverse objects and a pre-trained Res-Net (He et al, 2016 ) is utilized to accomplish object perception and feature extraction for querying grasp affordances by Gibbs sampling (Kim and Nelson, 1999 ). The main thought back Ardón et al ( 2019 ) is sampling several grasp affordances and evaluate them, the affordance with highest possibility will be selected and corresponding grasp configuration is calculated. Inspired by leveraging rectangle represent grasp part (Jiang et al, 2011 ; Lenz et al, 2015 ), Vohra et al ( 2019 ) and Yu Q. et al ( 2020 ) sample numerous rectangles to characterize candidate graspable parts and gain the optimal grasp pose by filtering and scoring candidates.…”

Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

“…Methods for learning relations between objects, between object properties, and between objects and their environments have shown to be beneficial for detecting objects on table tops [36,27,51], finding hidden objects in shelves [48], predicting object affordances [71], and semantic grasping [3,43]. However, most methods leverage probabilistic logic models to learn these relations, which have scalability issues that limit them from modeling inter-connected relations in larger domains [51,48,71,3]. In contrast, our proposed framework learns n-ary relations between 15 property types and 200 properties, the richest representation to date.…”

“…We apply this model to learn n-ary relations between object properties. • Markov Logic Network (MLN) represents probabilistic logic languages that have been used to model complex semantic relations in various robotic domains [51,71,52,3,13]. We closely follow prior work to specify probabilistic rules for our domain.…”

“…Semantic task and object knowledge serves as a valuable abstraction in this context. Perceiving and understanding semantic properties of objects (e.g., a cup is ceramic, empty, located in kitchen, and used for drinking) aids robots in performing many realworld tasks, such as inferring missing information in human instructions [52,12], efficiently searching for objects in homes [69,68], and manipulating objects based on their affordances and states [3,43,31].…”