This article presents a method of real-time robot-manipulator teleoperation using markerless image-based hand-arm tracking. The markerless tracking is carried out by processing images from two calibrated cameras to estimate in three dimensions (3-D), the positions of the wrist joint, elbow joint, index finger, and thumb. The hand pose (position and orientation) is used to specify the pose of the end-effector of a robot-manipulator in real-time teleoperation. The method permits a natural means of communicating an entire task to a robot, rather than using limited motion commands as with gesture-based approaches, and it has been demonstrated for pick-and-place tasks.
The control of a robot manipulator by a human operator is often necessary in unstructured dynamic environments with unfamiliar objects. Remote teleoperation is required when human presence at the robot site is undesirable or difficult, such as in handling hazardous materials and operating in dangerous or inaccessible environments. Previous approaches have employed mechanical or other contacting interfaces which require unnatural motions for object manipulation tasks or hinder dexterous human motion. This paper presents a non-contacting method of teleoperating a robot manipulator by having the human operator perform the three-dimensional (3D) human hand-arm motion that would naturally be used to complete an object manipulation task and tracking the motion with a stereo-camera system at a local site. The 3D human hand-arm motion is reconstructed at the remote robot site and is used to control the position and orientation of the robot manipulator end-effector in real-time. Images captured of the robot interacting with objects at the remote site provide visual feedback to the human operator. Tests in teleoperation of the robot manipulator have demonstrated the ability of the human to carry out object manipulation tasks remotely and the capability of the teleoperated robot manipulator system to copy human hand-arm motions in real-time.
In robot teleoperation, contacting mechanical devices and sensors have been commonly used to track operator hand and arm motion. While camera-based tracking has the benefit of being non-contacting, markerless camera-based human tracking offers a further advantage of not requiring markers and thus avoiding marker occlusion. This paper presents an application of markerless image-based arm tracking to real-time teleoperation of a robot manipulator. The markerless tracking is carried out by processing images from two calibrated cameras in real-time, to estimate the positions of the joint centres of the wrist and elbow in three dimensions (3D), and to compute the 3D positions of the index finger and thumb in order to estimate the hand orientation. These are used to determine the position and orientation of the endeffector of a robot-manipulator in real-time teleoperation. Markerless tracking for teleoperation was demonstrated for pick-and-place tasks.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.