Refining grasp affordance models by experience

Abstract

Abstract-We present a method for learning object grasp affordance models in 3D from experience, and demonstrate its applicability through extensive testing and evaluation on a realistic and largely autonomous platform. Grasp affordance refers here to relative object-gripper configurations that yield stable grasps. These affordances are represented probabilistically with grasp densities, which correspond to continuous density functions defined on the space of 6D gripper poses. A grasp density characterizes an object's grasp affordance; densities are linked to visual stimuli through registration with a visual model of the object they characterize. We explore a batch-oriented, experience-based learning paradigm where grasps sampled randomly from a density are performed, and an importance-sampling algorithm learns a refined density from the outcomes of these experiences. The first such learning cycle is bootstrapped with a grasp density formed from visual cues. We show that the robot effectively applies its experience by downweighting poor grasp solutions, which results in increased success rates at subsequent learning cycles. We also present success rates in a practical scenario where a robot needs to repeatedly grasp an object lying in an arbitrary pose, where each pose imposes a specific reaching constraint, and thus forces the robot to make use of the entire grasp density to select the most promising achievable grasp.

Object–Action Complexes: Grounded abstractions of sensory–motor processes

“…This OAC can be used to model reactive or affordance-based behaviours (see [42,30]) as outlined in Section 7.2.2. At an intermediate level in another grasp-related OAC (described in Section 7.3), grasp densities are used to hypothesise possible grasps when the agent has some object knowledge [44]. Finally, at the highest level, plans effectively use grasps to manipulate objects on an abstracted symbolic scene representation (see Section 7.4).…”

“…9 An object model includes a learnt, structural object model that represents geometric relations between 3D visual patches (i.e., early cognitive vision (ECV) features [54]) as Markov networks [55]. In addition, it contains a continuous representation of object-relative gripper poses that lead to successful grasps by means of grasp densities [44]. Object detection, pose estimation, and the determination of useful gripper poses for grasping the object are all done simultaneously using probabilistic inference within the Markov network, given a scene reconstruction in terms of ECV features.…”

“…d) Grasp hypothesis densities for specific objects such as the basket (right) are generated at uniform orientations around 3D ECV features (left). e) Empirical grasp density learnt by testing grasps drawn from a hypothesis density [44].…”

“…In this scenario, the scene contains up grasps drawn from g 0 (learning g 1 ) success rate Figure 9: Evolving statistics M of statusGrasp = stable for the OACs ObjGrasp Pan , ObjGrasp Knife , and ObjGrasp Basket over successive rounds of grasping trials [44]. In the first round, grasps are drawn from a hypothesis density g 0 generated from ECV features (Figure 8) for each object; the red bars show the empirical success rates, and the grasp density computed over the successful grasps is denoted g 1 .…”

Learning Adjectives and Nouns from Affordances on the iCub Humanoid Robot

“…Note that an agent would require more of such relations on different objects and behaviours to learn more general affordance relations and to conceptualize over its sensorimotor experiences. During the last decade, similar formalizations of affordances proved to be very practical with successful applications to domains such as navigation [15], manipulation [16,17,18,19,20], conceptualization and language [5,4], planning [18], imitation and emulation [12,18,4], tool use [21,22,13] and vision [4]. A notable one with a notion of affordances similar to ours is presented by Montesano et al [23,24].…”