Beyond the Self: Using Grounded Affordances to Interpret and Describe Others’ Actions

Abstract

We propose a developmental approach that allows a robot to interpret and describe the actions of human agents by reusing previous experience. The robot first learns the association between words and object affordances by manipulating the objects in its environment. It then uses this information to learn a mapping between its own actions and those performed by a human in a shared environment. It finally fuses the information from these two models to interpret and describe human actions in light of its own experience. In our experiments, we show that the model can be used flexibly to do inference on different aspects of the scene. We can predict the effects of an action on the basis of object properties. We can revise the belief that a certain action occurred, given the observed effects of the human action. In an early action recognition fashion, we can anticipate the effects when the action has only been partially observed. By estimating the probability of words given the evidence and feeding them into a pre-defined grammar, we can generate relevant descriptions of the scene. We believe that this is a step towards providing robots with the fundamental skills to engage in social collaboration with humans.

Artificial cognition vs. artificial intelligence for next-generation autonomous robotic agents

“…The actual developmental level characterizes mental development retrospectively, while the Zone of Proximal Development characterizes mental development prospectively. For example, we may quote a representative list of studies in the two preceding decades that cover in isolation some of the specific issues of developmental robotics: Self-exploration and early imitation (Kuniyoshi et al, 2003), Modeling joint attention (Nagai, 2007), Scaffolding Robot Action Learning (Nagai and Rohlfing, 2009), Affordance-based perception (Min et al, 2016), Bootstrapping the semantics of tools (Schoeler and Wörgötter, 2016), Perception of Localized Features (Giagkos et al, 2017), Bootstrapping of Sensory-Motor Skills (Wieser and Cheng, 2018), Developing Reaching Ability like human infants (Luo et al, 2018), Sensorimotor Communication (Donnarumma et al, 2012(Donnarumma et al, , 2018Pezzulo et al, 2019), Integration of Sensing, Cognition, Learning, and Control (Li et al, 2019), Evaluation of Internal Models (Smith and Herrmann, 2019), Emergence of symbolic representations (Ugur et al, 2015;Taniguchi et al, 2019), Grounded affordances (Saponaro et al, 2020), Bodily Expression of Emotion (Tuyen et al, 2021), Morphological development (Naya-Varela et al, 2021), Skill Learning Strategy with Dynamic Motion Primitives (Li et al, 2021), Interest-driven exploration (Rayyes et al, 2022(Rayyes et al, , 2023.…”