Early integration of vision and manipulation

Abstract

Vision and manipulation are inextricably intertwined in the primate brain. Tantalizing results from neuroscience are shedding light on the mixed motor and sensory representations used by the brain during reaching, grasping, and object recognition. We now know a great deal about whar happens in the brain during these activities. but not necessarily why. Is the integration we , see functionally important. or just a reflection of evolution's lack of enthusiasm for sharp modularity? We wish to instantiate these results in robotic form to probe their technical advantages and to find any lacunae in existing models. We believe it would he missing the point to investigate this on a platform where dextrous manipulation and sophisticated machine vision are already implemented in their mature form, and instead follow a developmental approach from simpler primitives.We begin with a precursor to manipulation, simple poking and prodding, and show how it facilitates object segmentation, a long-standing problem in machine vision. The robot can familiarize itself with the objects in its environment by acting upon them. It can then recognize other actors (such as humans) in the environment through their effect on the objects it has leaned about. We argue that following causal chains of events out from the robot's body into the environment allows for a very natural developmental progression of visual competence, and relate this idea to results in neuroscience.

Cognitive Systems Platforms using Open Source

“…Following the object behavior, the association of sensory and motor cues allows the autonomous acquisition of object affordances [47]. Object affordances are a powerful tool for a cognitive system, allowing the interpretation of scenes (because of the behavior of objects) as well as the imitation of other people's actions (as a consequence of the behavior of objects) [48].…”

Active exploration and keypoint clustering for object recognition

“…Firstly, we use it to separate the object from its background, similar to [5,16]. We use a method that can be described as what-moves-together-belongstogether.…”

“…thermore, exploration of the object makes it possible to separate the object from its background, something that is nontrivial when passively observing an object on a highly cluttered background [16].…”

Teaching a Robotic Child - Machine Learning Strategies for a Humanoid Robot from Social Interactions

“…Infant development research suggests ways to acquire such training data from simple contexts, and use these experiences to bootstrap to more complex contexts. We need to identify situations that enable the robot to temporarily reach beyond its current perceptual abilities (Figure 2 shows a set of such situations), giving the opportunity for development to occur (Arsenio, 2004c;d;Metta & Fitzpatrick, 2003). This led us to create children-like learning scenarios for teaching a humanoid robot.…”

“…Tools developed to acquire such data, such as object / scene /texture segmentation (Arsenio, 2004e), sound / face segmentation (Arsenio, 2004f), object / face / hand tracking (Arsenio, 2004e;f) or even robot actuation for active perception (Arsenio, 2003;Metta & Fitzpatrick, 2003) are described in the literature. We do not treat children as machines, i.e., automatons.…”