DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System

Handa, Ankur; Wyk, Karl Van; Yang, Wei; Liang, Jacky; Chao, Yu-Wei; Wan, Qian; Birchfield, Stan; Ratliff, Nathan; Fox, Dieter

doi:10.1109/icra40945.2020.9197124

Cited by 111 publications

(98 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the delay of the robot causes some sense of contour in the actual running trajectory, the robot can follow the local motion well on the whole. In comparison, whole-process location mapping [38] requires a user to operate all trajectories, including movement, grasping, obstacle avoidance, and placement, which is a heavy burden on the user. In some situations, automatic operation can be achieved, such as peg-in-hole assembly [39] with complex requirements.…”

Section: Resultsmentioning

confidence: 99%

Gesture- and vision-based automatic grasping and flexible placement in teleoperation

Zhao

Chen

et al. 2022

Int J Adv Manuf Technol

View full text Add to dashboard Cite

Teleoperation system has attracted a lot of attention because of its advantages in dangerous or unknown environment.It is very difficult to develop an operating system that can complete complex tasks in an completely autonomous. This paper proposes a robot arm control strategy based on gesture and visual perception. The strategy combines the advantages of humans and robots to obtain a convenient and flexible interaction model. The hand data were obtained by Leap-Motion. Then a neural network algorithm was used to classify the nine gestures used for robot control by a finite state machine. The control mode switched between indicative control and mapping control. The robot acquired a autonomous grasp ability by incorporating YOLO 6D, depth data, and a probabilistic roadmap planner algorithm.The robot completed most of the trajectory independently, and a few flexible trajectories required a user to make mapping actions. This interactive mode reduces the burden of the user to a certain extent, that makes up for the shortcomings of traditional teleoperation.

show abstract

Section: Resultsmentioning

confidence: 99%

Gesture- and vision-based automatic grasping and flexible placement in teleoperation

Zhao

Chen

et al. 2022

Int J Adv Manuf Technol

View full text Add to dashboard Cite

show abstract

“…In addition, as we had the inverse kinematics for the delta robots, we were able to quickly translate a desired trajectory into commands to the linear actuators. This direct mapping allowed us to easily teleoperate the robot with a PS4 Controller to complete tasks that would typically require a motion tracking hand setup in order to give the robot demonstrations [21]. In the future, we plan to explore more delicate and dexterous tasks with added sensors to provide feedback when interacting with objects.…”

Section: Methodsmentioning

confidence: 99%

A Low-Cost Compliant Gripper Using Cooperative Mini-Delta Robots for Dexterous Manipulation

Mannam¹,

Rudich²,

Zhang³

et al. 2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

Traditional parallel-jaw grippers are insufficient for delicate object manipulation due to their stiffness and lack of dexterity. Other dexterous robotic hands often have bulky fingers, rely on complex time-varying cable drives, or are prohibitively expensive. In this paper, we introduce a novel low-cost compliant gripper with two centimeter-scaled 3-DOF delta robots using offthe-shelf linear actuators and 3D-printed soft materials. To model the kinematics of delta robots with soft compliant links, which diverge from typical rigid links, we train neural networks using a perception system. Furthermore, we analyze the delta robot's force profile by varying the starting position in its workspace and measuring the resulting force from a push action. Finally, we demonstrate the compliance and dexterity of our gripper through six dexterous manipulation tasks involving small and delicate objects. Thus, we present the groundwork for creating modular multi-fingered hands that can execute precise and lowinertia manipulations.

show abstract

“…Recent advances in deep learning-based machine perception have substantially improved the accuracies of pose estimation from real-world sensory data [2][3][4][5][6]. The estimated 6-DoF object pose, represented by the translation and rotation in SE(3), serves as a compact and informative state representation for a variety of downstream tasks, such as robot grasping and manipulation [7], human-robot interactions [8], online camera calibration [9], and tele-presence robot control [10].…”

Section: Introductionmentioning

confidence: 99%

Fast Uncertainty Quantification for Deep Object Pose Estimation

Shi

Zhu

Tremblay³

et al. 2021

2021 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

Deep learning-based object pose estimators are often unreliable and overconfident especially when the input image is outside the training domain, for instance, with sim2real transfer. Efficient and robust uncertainty quantification (UQ) in pose estimators is critically needed in many robotic tasks. In this work, we propose a simple, efficient, and plug-and-play UQ method for 6-DoF object pose estimation. We ensemble 2-3 pre-trained models with different neural network architectures and/or training data sources, and compute their average pairwise disagreement against one another to obtain the uncertainty quantification. We propose four disagreement metrics, including a learned metric, and show that the average distance (ADD) is the best learning-free metric and it is only slightly worse than the learned metric, which requires labeled target data. Our method has several advantages compared to the prior art: 1) our method does not require any modification of the training process or the model inputs; and 2) it needs only one forward pass for each model. We evaluate the proposed UQ method on three tasks where our uncertainty quantification yields much stronger correlations with pose estimation errors than the baselines. Moreover, in a real robot grasping task, our method increases the grasping success rate from 35% to 90%. Video and code are available at https://sites.google.com/view/fastuq.

show abstract

DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System

Cited by 111 publications

References 37 publications

Gesture- and vision-based automatic grasping and flexible placement in teleoperation

Gesture- and vision-based automatic grasping and flexible placement in teleoperation

A Low-Cost Compliant Gripper Using Cooperative Mini-Delta Robots for Dexterous Manipulation

Fast Uncertainty Quantification for Deep Object Pose Estimation

Contact Info

Product

Resources

About