PARTI-A Haptic Virtual Reality Control Station for Model-Mediated Robotic Applications

Abstract

In this paper, we introduce a tele-robotic station called “PARTI” that leverages state-of-the-art robotics and virtual reality technology to enable immersive haptic interaction with simulated and remote environments through robotic avatars. Our hardware-in-the-loop framework integrates accurate multibody system dynamics and frictional contacts with digital twins of our robots in a virtual environment with real-time computational capabilities. This model mediated hardware-in-the-loop approach to robotic control allows a teleoperator to use the PARTI system to teach, evaluate, and control various robotic applications. In the current contribution, we focus on the general system description, integrated simulation and control framework, and a series of experiments highlighting the advantages of our approach.

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

“…With the advancement of data-driven approaches for robot manipulation [12,18], there is a growing need to collect human demonstrations in robotics. To enable easy and scalable data collection, teleoperation has also gained attention in simulated environments [60,32,56,14,8].…”

“…Teleoperation [39], as a direct means to acquire human demonstrations for teaching robots, has been a powerful paradigm to approach this goal [22,11,64,17,34,6,19,5,38,51]. Compared to gripper-based manipulators, teleoper-ating dexterous hand-arm systems poses unprecedented challenges and often requires specialized apparatus that comes with high costs and setup efforts, such as Virtual Reality (VR) devices [4,17,15], wearable gloves [29,30], handheld controller [45,46,20], haptic sensors [12,23,50,53], or motion capture trackers [65]. Fortunately, recent developments in vision-based teleoperation [2,24,16,26,42,27,21,22,3] have provided a low-cost and more generalizable alternative for teleoperating dexterous robot systems.…”

DexFuncGrasp: A Robotic Dexterous Functional Grasp Dataset Constructed from a Cost-Effective Real-Simulation Annotation System

“…For the high-DoF dexterous robot hand dataset annotation system, it is supposed to be as efficient as possible. Previous works commonly use wearable gloves (Liu et al 2017(Liu et al , 2019a or haptic sensors (Elsner et al 2022;Kumar and Todorov 2015) to collect robot actions from human demonstrations, however, the high costs limit their wide…”