Situated robot learning for multi-modal instruction and imitation of grasping

Abstract

A key prerequisite to make user instruction of work tasks by interactive demonstration effective and convenient is situated multi-modal interaction aiming at an enhancement of robot learning beyond simple low-level skill acquisition. We report the status of the Bielefeld GRAVIS-robot system that combines visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation to allow multi-modal task-oriented instructions. With respect to this platform, we discuss the essential role of learning for robust functioning of the robot and sketch the concept of an integrated architecture for situated learning on the system level. It has the long-term goal to demonstrate speech-supported imitation learning of robot actions. We describe the current state of its realization to enable imitation of human hand postures for flexible grasping and give quantitative results for grasping a broad range of everyday objects.

A Dual Interaction Perspective for Robot Cognition: Grasping as a “Rosetta Stone”

“…Based on these very encouraging results, we are currently porting the capabilities of a previously developed system with a more limited robot manipulator, but coupled with speech-understanding and binocular vision capabilities [2,6,7], to the new architecture. Developed over a time horizon of several years, this previous system had reached the limits of its extensibility and maintainability.…”

“…Empirically we found, that by careful finetuning of the four employed grasp prototypes, the described grasping architecture can capture the relevant "reactive grasp knowledge" sufficiently well to allow successful grasps for a considerable range of everyday objects [6]. Many alternative approaches to grasping put a high emphasis on rather elaborate grasp planning [12,13,14], achieved with sophisticated optimisation methods to find optimal contact points [15,16]), and with very precise joint position control.…”

“…A few examples of such systems acting in a robotics or pure vision domain can be found in [2,3,4,5]. Some of our own earlier work along these lines is summarised in [6,7]. Experience from this work led to the dual interaction perspective described above and to ideas for its implementation in a compact and highly manageable architecture employing hierarchical state machines (HSMs).…”

Platform portable anthropomorphic grasping with the bielefeld 20-DOF shadow and 9-DOF TUM hand

“…Consequently, we have previously proposed to enable imitation grasping [18] in the context of a long-term research project [19] aiming at the realization of a robot system that is instructable by speech and gesture, has visual capabilities, attentive behavior, and can execute grasping actions [20], [21] (see Fig. 1).…”

“…In this framework, we use a universal, biologically motivated grasp strategy, which relies on a 3D localization of the object, executes a reaching movement, and finally grasps an object employing appropriate pregrasp and target grasp postures. In [18], we have proposed an object-specific grasp selection based on the observation of a human instructor's hand to reduce complexity of the selection process and enhance grasp success. The present paper focuses on comparative results obtained with a new, much more dextrous anthropomorphic hand containing 24 joints actuated by pneumatic muscles.…”