Leveraging depth data in remote robot teleoperation interfaces for general object manipulation

Abstract

Robust remote teleoperation of high-degree-of-freedom manipulators is of critical importance across a wide range of robotics applications. Contemporary robot manipulation interfaces primarily utilize a free positioning pose specification approach to independently control each degree of freedom in free space. In this work, we present two novel interfaces, constrained positioning and point-and-click. Both novel approaches incorporate scene information from depth data into the grasp pose specification process, effectively reducing the number of 3D transformations the user must input. The novel interactions are designed for 2D image streams, rather than traditional 3D virtual scenes, further reducing mental transformations by eliminating the controllable camera viewpoint in favor of fixed physical camera viewpoints. We present interface implementations of our novel approaches, as well as free positioning, in both 2D and 3D visualization modes. In addition, we present results of a 90-participant user study evaluation comparing the effectiveness of each approach for a set of general object manipulation tasks, and the effects of implementing each approach in 2D image views versus 3D depth views. The results of our study show that point-and-click outperforms both free positioning and constrained positioning by significantly increasing the number of tasks completed and significantly reducing task failures and grasping errors, while significantly reducing the number of user interactions required to specify poses. In addition, we found that regardless of the interaction approach, the 2D visualization mode resulted in significantly better performance than the 3D visualization mode, with statistically significant reductions in task failures, grasping errors, task completion time, number of interactions, and user workload, all while reducing bandwidth requirements imposed by streaming depth data.

A Review of NASA Human-Robot Interaction in Space

“…Chernova et al have developed remote manipulation interfaces that are robust to high latency environments, researching how to leverage depth data in remote robot teleoperation interfaces for general object manipulation [43,44] and developing temporal models for robot classification of human interruptibility for modeling availability of collocated human crew members [45].…”

Advanced teleoperation and control system for industrial robots based on augmented virtuality and haptic feedback

“…If a 2D camera is used, the depth can be estimated by processing the acquired data [35]. Alternatively, this estimation is not needed if a 3D camera is used [36], e.g., the Microsoft Kinect.…”

Task-Level Authoring for Remote Robot Teleoperation

“…In this case, the robot can run parameterized subroutines while multi-person teams of highly trained operators analyze data from the robot and control it at various abstraction-levels (from joint angle to locomotion goal), including situations with unstable communication channels ( Johnson et al, 2015 ). These subroutines can be parameterized by selecting or moving virtual markers displaying the grasping pose ( Kent et al, 2020 ), robot joint position ( Nakaoka et al, 2014 ), or using affordance templates ( Hart et al, 2014 ). In a retrospective analysis, Yanco et al (2015) highlight the training required for operating the robots during these trials, and reports that researchers should explore new interaction methods that could be used by first responders without extensive training.…”