A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot

Abstract

Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captures RGB-D data that is streamed to a SLAM-based live multiclient telepresence system. Here, a global 3D model of the already captured scene parts is reconstructed and streamed to the individual remote user clients where the rendering for e.g. head-mounted display devices (HMDs) is performed. We introduce a novel lightweight robot client component which transmits robot-specific data and enables a quick integration into existing robotic systems. This way, in contrast to firstperson exploration systems, the operators can explore and navigate in the remote site completely independent of the current position and view of the capturing robot, complementing traditional input devices for teleoperation. We provide a proofof-concept implementation and demonstrate the capabilities as well as the performance of our system regarding interactive object measurements and bandwidth-efficient data streaming and visualization. Furthermore, we show its benefits over purely video-based teleoperation in a user study revealing a higher degree of situation awareness and a more precise navigation in challenging environments.

Intuitive Robot Teleoperation Through Multi-Sensor Informed Mixed Reality Visual Aids

“…We find very appropriate the suggested use of a small number of tasks, and we wish these to follow the order given by the MANTRA as it fits nicely to robot teleguide actions. We always Overview first, while Zooming, Filtering and Details are applied on demand [31] [35]. Automatic adaptation based on speed and distance to objects is an available option useful for specific environments and situations.…”

“…Rather, we focused on evaluating the interface capability of delivering multiple inputs to operators (from GPS, IMU, Camera and 3D graphic model). We had an Internet connection with delays ranging from 1 to 4 secs, which is in line with several works in the literature [7] [31]. We exploited the data-associated timestamps in order to serve all inputs and response on first-in first-served schedule.…”

“…It includes an imaginary bonnet shape (with pointed end) to help operators comprehend vehicle-direction. Viewing vehicle's front bonnet effectively increases situational awareness [2][31], and helps preventing collisions[5].…”

Human Perception-Optimized Planning for Comfortable VR-Based Telepresence

“…Heshmat et al [8] compared telepresence between a 2-D screen and an HMD, and found that people prefer the ability to look around with the HMD. Zhang et al [37] researched the use of redirected walking when using an HMD for teleoperation, and Stotko et al [30] built a model from the environment to be observed by the user while the robot moved. However, in [30], [37] the issue of VR sickness was avoided by using less scalable user interfaces, and in [8], [23] it was not considered at all.…”

Design of a Hyper-Redundant Robot and Teleoperation Using Mixed Reality for Inspection Tasks

“…Adaptive and immersive interfaces have shown improvements in situational awareness, workload and performance of operators that manage multi-robot systems with both aerial, ground and manipulator robots [32]. Other recent work demonstrated the feasibility of reconstructing scenarios with RGB-D sensors and, at the same time, operating robots through VR head-mounted devices [33]. Others used Augmented Reality to teleoperate an industrial robot for assembly tasks, increasing the accuracy, efficiency and reducing the differences of skillfulness [34].…”