3D stereo estimation and fully automated learning of eye-hand coordination in humanoid robots

Fanello, Sean; Pattacini, Ugo; Gori, Ilaria; Tikhanoff, Vadim; Randazzo, Marco; Roncone, Alessandro; Odone, Francesca; Metta, Giorgio

doi:10.1109/humanoids.2014.7041491

Cited by 32 publications

(27 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Open-loop approach: before entering the visual servoing control loop, we use the iCub stereo vision to get a rough 3D localization of our target. In particular, we employ a Structure From Motion algorithm [30] to get such 3D point and then we move the right hand using an open loop control. Observe goal and end effector: in order to carry out visual tracking and visual servoing the robot has to observe both the target object and its end-effector.…”

Section: A Experimental Setupmentioning

confidence: 99%

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

Fantacci

Pattacini

Tikhanoff

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Abstract-This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images are then used to extract information and estimate the pose of the end-effector. To this aim, we developed a particle filter for estimating the position and orientation of the robot's end-effector using the Histogram of Oriented Gradient (HOG) descriptors to capture robust characteristic features of shapes in both cameras and rendered images. We implemented the algorithm on the iCub humanoid robot and employed it in a closed-loop reaching scenario. We demonstrate that the tracking is robust to clutter, allows compensating for errors in the robot kinematics and servoing the arm in closed loop using vision.

show abstract

Section: A Experimental Setupmentioning

confidence: 99%

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

Fantacci

Pattacini

Tikhanoff

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…Although it seems that no markers are used, no description is present about how the end-effector pose is measured. In Fanello et al (2014), eyehand calibration is realized by performing several ellipsoidal arm movements with a predefined hand posture, tracking the tip of the index finger in the camera images. Optimization techniques are employed to learn the transformation between the fingertip position obtained by the stereo vision and the one computed from the forward kinematics.…”

Section: Related Workmentioning

confidence: 99%

Online Body Schema Adaptation Based on Internal Mental Simulation and Multisensory Feedback

2016

View full text Add to dashboard Cite

In this paper, we describe a novel approach to obtain automatic adaptation of the robot body schema and to improve the robot perceptual and motor skills based on this body knowledge. Predictions obtained through a mental simulation of the body are combined with the real sensory feedback to achieve two objectives simultaneously: body schema adaptation and markerless 6D hand pose estimation. The body schema consists of a computer graphics simulation of the robot, which includes the arm and head kinematics (adapted online during the movements) and an appearance model of the hand shape and texture. The mental simulation process generates predictions on how the hand will appear in the robot camera images, based on the body schema and the proprioceptive information (i.e., motor encoders). These predictions are compared to the actual images using sequential Monte Carlo techniques to feed a particle-based Bayesian estimation method to estimate the parameters of the body schema. The updated body schema will improve the estimates of the 6D hand pose, which is then used in a closed-loop control scheme (i.e., visual servoing), enabling precise reaching. We report experiments with the iCub humanoid robot that support the validity of our approach. A number of simulations with precise ground-truth were performed to evaluate the estimation capabilities of the proposed framework. Then, we show how the use of high-performance GPU programing and an edge-based algorithm for visual perception allow for real-time implementation in real-world scenarios.

show abstract

“…However, the use of depth estimation as described above in real-world scenarios is hindered in the earlier methods 28,29 by the difficulty of computing realtime and robust disparity maps from moving stereo cameras. In these scenarios, the proposed depth estimation techniques help the robot to identify the ball and environment and prepare the robot for next action in the real-world settings.…”

Section: Trigonometric Depth Estimationmentioning

confidence: 99%

“…In these scenarios, the proposed depth estimation techniques help the robot to identify the ball and environment and prepare the robot for next action in the real-world settings. Regarding the estimation of the camera parameters, the procedure described in the studies by Fanello et al 28 and Ciliberto et al 29 was adapted except the calibration procedures. The proposed method is a viable as compared with articles referred in the studies by Pasquale et al 30 and Sadeh-Or and Kaminka 31 in dynamical conditions as these pose static constraints in real-world scenarios.…”

Section: Trigonometric Depth Estimationmentioning

confidence: 99%

Behaviour coordinations and motion synchronizations for humanoid robot

Parasuraman

Hock

Khan

et al. 2017

International Journal of Advanced Robotic Systems

View full text Add to dashboard Cite

Many features have to be solved by humanoid robot during soccer game to get evidences from the environment such as detect ball, goal, lines and other robotmates. Having these data, the robot has to self-localize and proceed for next action reactively and ensure sense-think-act process efficiently. Sense-think-act processes are still a challenge task for humanoid robots. Hence, a modular framework is proposed for soccer ball game in which the architecture is mainly composed of object detection, field detection and motion synchronization behaviours. Object detection is modularized into ball detection, segmentation and depth estimation to facilitate the control actions. Similarly, field detection is modularized into goalpost and boundaries detection. Motion synchronization is modularized into primitives such as scoring, kip up and diving which uses the proposed support polygon and centre of moment methods. The behaviour synchronization and execution takes place in multilayers which include player and keeper mode as expert layer, modular behaviours as reactive layers and servo and motor command are executed in skill layer. The behaviour analysis and performance are targeted on the trigonometric depth estimation, grid-based segmentation pattern learning and recognition as well as support polygon and Centre Of Mass (COM). Experimental results are demonstrated and discussed. The proposed modular framework in this work has been tested using the NAO robot.

show abstract

3D stereo estimation and fully automated learning of eye-hand coordination in humanoid robots

Cited by 32 publications

References 25 publications

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

Online Body Schema Adaptation Based on Internal Mental Simulation and Multisensory Feedback

Behaviour coordinations and motion synchronizations for humanoid robot

Contact Info

Product

Resources

About