Linking language with embodied and teleological representations of action for humanoid cognition

Abstract

The current research extends our framework for embodied language and action comprehension to include a teleological representation that allows goal-based reasoning for novel actions. The objective of this work is to implement and demonstrate the advantages of a hybrid, embodied-teleological approach to action–language interaction, both from a theoretical perspective, and via results from human–robot interaction experiments with the iCub robot. We first demonstrate how a framework for embodied language comprehension allows the system to develop a baseline set of representations for processing goal-directed actions such as “take,” “cover,” and “give.” Spoken language and visual perception are input modes for these representations, and the generation of spoken language is the output mode. Moving toward a teleological (goal-based reasoning) approach, a crucial component of the new system is the representation of the subcomponents of these actions, which includes relations between initial enabling states, and final resulting states for these actions. We demonstrate how grammatical categories including causal connectives (e.g., because, if–then) can allow spoken language to enrich the learned set of state-action-state (SAS) representations. We then examine how this enriched SAS inventory enhances the robot's ability to represent perceived actions in which the environment inhibits goal achievement. The paper addresses how language comes to reflect the structure of action, and how it can subsequently be used as an input and output vector for embodied and teleological aspects of action.

When shared plans go wrong: From atomic- to composite actions and back

“…In particular, the ability of children to take a 'birdseye-view' and understand their and their partners actions as part of a shared plan [8] is a concept we have recently implemented in several robotic systems [9], [10], [11]. Similar work by Shah et al [12], has recently demonstrated that human-robot team performance is improved by shared planning in comparison to robots that are verbally commanded by the human.…”

The Coordinating Role of Language in Real-Time Multimodal Learning of Cooperative Tasks

“…Yet the true notion of the actual final goal, the shared intention, to get that toy into the box, is currently not present. We have started to address this issue by linking actions to their resulting states, within the action representation [56]. We must go further, in order to now expand the language capability to address the expression and modification of internal representations of the intentional states of others.…”

“…John gave Sally a flower), Subject maps onto the agent of the transitive action specified by Verb, and Recipient receives the Object via that transitive action. The current research demonstrates how language, based on these constructions, can be used to coordinate real-time learning of cooperative actions, providing the coordination of multiple demonstration modalities including vision-like perception, kinesthetic demonstration [13,29,[55][56][57][58], and command execution via spoken language. In this sense, language serves a dual purpose: First and most important, it provides the mechanism by which a cooperative plan can be constructed and modified.…”

NARLE: Neurocognitive architecture for the autonomous task recognition, learning, and execution

“…Learning from demonstration (Argall, Chernova, Veloso, & Browning, 2009) and imitation learning of movement primitives (Ijspert, Nakanishi, & Schaal, 2002;Mochizuki, Nishide, Okuno, & Ogata, 2013) are among the most successful attempts to implement robotic learning systems. Using insights from work on human-robot cooperation (Dominey, Mallet, & Yoshida, 2009;Lallee, Madden, Hoen, & Dominey, 2010; and segmentation of observed action sequences (GuerraFilho & Aloimonos, 2006;Wörgötter et al, 2013), in our work, we aim at more ''cognitive'' actions, which not only specify the desired trajectory of the robot's effectors, but also the properties of the target object and its desired final state (Cuijpers, Schie, Koppen, Erlhagen, & Bekkering, 2006). Learning such goal-directed behaviors and learning sequence of such behaviors simultaneously is an open problem.…”