Online Body Schema Adaptation Based on Internal Mental Simulation and Multisensory Feedback

Abstract

In this paper, we describe a novel approach to obtain automatic adaptation of the robot body schema and to improve the robot perceptual and motor skills based on this body knowledge. Predictions obtained through a mental simulation of the body are combined with the real sensory feedback to achieve two objectives simultaneously: body schema adaptation and markerless 6D hand pose estimation. The body schema consists of a computer graphics simulation of the robot, which includes the arm and head kinematics (adapted online during the movements) and an appearance model of the hand shape and texture. The mental simulation process generates predictions on how the hand will appear in the robot camera images, based on the body schema and the proprioceptive information (i.e., motor encoders). These predictions are compared to the actual images using sequential Monte Carlo techniques to feed a particle-based Bayesian estimation method to estimate the parameters of the body schema. The updated body schema will improve the estimates of the 6D hand pose, which is then used in a closed-loop control scheme (i.e., visual servoing), enabling precise reaching. We report experiments with the iCub humanoid robot that support the validity of our approach. A number of simulations with precise ground-truth were performed to evaluate the estimation capabilities of the proposed framework. Then, we show how the use of high-performance GPU programing and an edge-based algorithm for visual perception allow for real-time implementation in real-world scenarios.

Robot Self-Calibration Using Multiple Kinematic Chains—A Simulation Study on the iCub Humanoid Robot

“…Hersch et al [7] and Martinez-Cantin et al [8] present online methods to calibrate humanoid torso kinematics relying on gradient descent and recursive least squares estimation, respectively. The iCub humanoid was employed in [9], [4]. Vicente et al [9] used a model of the hand's appearance to estimate its 6D pose and used that information to calibrate the joint offsets.…”

“…The iCub humanoid was employed in [9], [4]. Vicente et al [9] used a model of the hand's appearance to estimate its 6D pose and used that information to calibrate the joint offsets. Fanello et al [4] had the robot observe its fingertip and learned essentially a single transformation only to account for the discrepancy between forward kinematics of the arm and the projection of the finger into the cameras.…”

Markerless Eye-Hand Kinematic Calibration on the iCub Humanoid Robot

“…In the function initSMC we initialize the variables of the Sequential Monte Carlo parameter estimation, i.e., the initial distribution p(β 0 ) [Eq. 10in Vicente et al (2016a)], and the initial artificial dynamic noise. The Listings 1 contains the initSMC function where some of the variables (in red) are parametrized at initialization time (check sub-section 5.2.1 for more details on the initialization parameters).…”

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

“…This is fundamental for grasping objects. The first breakthrough dealing with humanoid robots and specifically tracking of an anthropomorphic hand are relatively recent, dating back to 2006 [29]- [32].…”

“…In [31], [32], a computer graphics simulator is exploited to create the body schema of the robot, including an appearance model of the hand shape with texture. The output of the simulator is used to generate predictions about hand appearance in the robot camera images, based on the sensorimotor proprioceptive information, i.e.…”