Accelerating Interactive Human-like Manipulation Learning with GPU-based Simulation and High-quality Demonstrations

Mosbach, Malte; Moraw, Kara; Behnke, Sven

doi:10.1109/humanoids53995.2022.10000161

Cited by 2 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It relies on accurate tracking of human hand motions and finger articulations. Compared to the costly wearable hand tracking solutions, such as gloves [29,30,37], marker-based motion capture systems [65,31], inertia sensors [63] or VR headsets [4,28,49,41], vision-based hand tracking is particularly favorable due to its low cost and low intrusion to the human operator. Early research in vision-based teleoperation focused on improving the performance of hand tracking [55,58,1] and mapping human hand pose to robot hand pose [24,2,5,35].…”

Section: Related Workmentioning

confidence: 99%

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

Qin,

Yang,

Huang

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

IsaacGym Simulator Real World* Yuzhe Qin was an intern at NVIDIA during the project.experiments, AnyTeleop can outperform a previous system that was designed for a specific robot hardware with a higher success rate, using the same robot. For teleoperation in simulation, AnyTeleop leads to better imitation learning performance, compared with a previous system that is particularly designed for that simulator.

show abstract

Section: Related Workmentioning

confidence: 99%

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

Qin,

Yang,

Huang

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

show abstract

Learning Generalizable Tool Use with Non-Rigid Grasp-Pose Registration

Mosbach,

Behnke

2023

2023 IEEE 19th International Conference on Automation Science and Engineering (CASE)

Self Cite

View full text Add to dashboard Cite

Interactive grasping from clutter, akin to human dexterity, is one of the longest-standing problems in robot learning. Challenges stem from the intricacies of visual perception, the demand for precise motor skills, and the complex interplay between the two. In this work, we present Teacher-Augmented Policy Gradient (TAPG), a novel two-stage learning framework that synergizes reinforcement learning and policy distillation.After training a teacher policy to master the motor control based on object pose information, TAPG facilitates guided, yet adaptive, learning of a sensorimotor policy, based on object segmentation. We zero-shot transfer from simulation to a real robot by using Segment Anything Model for promptable object segmentation. Our trained policies adeptly grasp a wide variety of objects from cluttered scenarios in simulation and the real world based on human-understandable prompts. Furthermore, we show robust zero-shot transfer to novel objects. Videos of our experiments are available at https://maltemosbach. github.io/grasp_anything.

show abstract

Accelerating Interactive Human-like Manipulation Learning with GPU-based Simulation and High-quality Demonstrations

Cited by 2 publications

References 16 publications

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

Learning Generalizable Tool Use with Non-Rigid Grasp-Pose Registration

Contact Info

Product

Resources

About