Virtual Reality based Telerobotics Framework with Depth Cameras

Abstract

This work describes a virtual reality (VR) based robot teleoperation framework which relies on scene visualization from depth cameras and implements human-robot and human-scene interaction gestures. We suggest that mounting a camera on a slave robot's end-effector (an in-hand camera) allows the operator to achieve better visualization of the remote scene and improve task performance. We compared experimentally the operator's ability to understand the remote environment in different visualization modes: single external static camera, in-hand camera, in-hand and external static camera, in-hand camera with OctoMap occupancy mapping. The latter option provided the operator with a better understanding of the remote environment whilst requiring relatively small communication bandwidth. Consequently, we propose suitable grasping methods compatible with the VR based teleoperation with the in-hand camera. Video demonstration: https://youtu.be/3vZaEykMS_E.

Automatic Fracture Characterization Using Tactile and Proximity Optical Sensing

“…Future research will focus on integrating a multi-modal approach with visual patches and implementation of the proposed system on a teleoperated mobile manipulation system. We plan to demonstrate how automatic fracture characterization will be efficiently integrated with the mobile manipulator controller (Farkhatdinov and Ryu, 2008 ) and how the obtained tactile data can be visualized in a dedicated virtual reality-based human-operator interface (Omarali et al, 2020 ). Further studies will be performed on dimensionality reduction with principal component analysis which may increase the classification accuracy.…”

Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

“…Augmented remote operation, or Augmented Telepresence (AT) [14], denotes applications where video-mediated communication is the enabling technology, but where additional data can be superimposed on or merged with the captured camera view as in AR. Such augmentation can be achieved on-site via using a see-through display and rendering only 3-Dimensional (3D) approximations of specific scene elements [15], or off-site via partial and full view rendering [16,17,9] for non-transparent displays. Unlike computer-generated imagery, this view rendering is at least partly based on the content of camera views.…”

“…View enhancement, or augmentation, is what distinguishes augmented remote operation from conventional VR headset based remote operation. Bejczy et al [10], Yun et al [5] and Omarali et al [16] are examples of conventional and augmented operation, all aiming to improve the remote control of a robotic arm. In [10,5], camera views of the scene are rendered to virtual display panels in a VR environment, without any change to the camera view content.…”

“…In [10,5], camera views of the scene are rendered to virtual display panels in a VR environment, without any change to the camera view content. In contrast, in [16], the camera views are fully replaced with a colored 3D point cloud model of the scene. The model is partly generated from the camera views, turning it into an extreme kind of view augmentation.…”

“…Disparity from a stereo camera is used to create the 3D structure of the scene. The 3D structure is shown in a separate view similar to [16,17], but it is also rotated and overlaid onto a 2D camera view, where false-color of the 3D model represents each pixel's depth value in the camera view. Each of these examples constructs an augmented view for the operator.…”