Mixed Reality Human–Robot Interface With Adaptive Communications Congestion Control for the Teleoperation of Mobile Redundant Manipulators in Hazardous Environments

Abstract

Robotic interventions with redundant mobile manipulators pose a challenge for telerobotics in hazardous environments, such as underwater, underground, nuclear facilities, particle accelerators, aerial or space. Communication issues can lead to critical consequences, such as imprecise manipulation resulting in collisions, breakdowns and mission failures. The research presented in this paper was driven by the needs of a real robotic intervention scenario in the Large Hadron Collider (LHC) at the European Organization for Nuclear Research (CERN). The goal of the work was to develop a framework for network optimisation in order to help facilitate Mixed Reality techniques such as 3D collision detection and avoidance, trajectories planning, real-time control, and automatized target approach. The teleoperator was provided with immersive interactions while preserving precise positioning of the robot. These techniques had to be adapted to delays, bandwidth limitation and their volatility in the 4G shared network of the real underground particle accelerator environment. The novel application-layer congestion control with automatic settings was applied for video and point cloud feedback. Twelve automatic setting modes were proposed with algorithms based on the camera frame rate, resolution, point cloud subsampling, network round-trip time and throughput to bandwidth ratio. Each mode was thoroughly characterized to present its specific use-case scenarios and the improvements it brings to the adaptive camera feedback control in teleoperation. Finally, the framework was presented according to which designers can optimize their Human-Robot Interfaces and sensor feedback depending on the network characteristics and task.

Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

“…Moreover, to facilitate multi-user collaboration and spatial feedback from the robot, an even larger amount of data must be continuously sent from the robot to the operators. In this paper, we measured what was the required communication load to provide useful feedback for a multi-user and multi-camera AR context and tested the proposed architecture of the com-munication system with the Adaptive Communications Congestion Control [10]. This publication extends the experimental results described in the previous publication.…”

“…• The safety of the operation must be assured by providing reliable hardware solutions and by creating intuitive interfaces that take profit of the robot's perception and display the synthesized information. • Up to now, the operational CERN human-robot interfaces used screens for visualization and keyboard, gamepad, space mouse, or a master-slave control with haptic feedback for control [10] [11] [22]. However, in a complex teleoperated system extended mapping of key bindings and actions are difficult to remember, and a more intuitive input system should be designed.…”

“…• Results of tasks execution in single and multi-operator scenarios, operators' feedback, and network performance are presented in Section IV and discussed in Section V. • Section VI concludes the findings and proposes further work. Moreover, this paper should be read together with our previous publication [10], as it references its multiple sections and figures. Also, the experiments performed here use several scenarios the Adaptive Communications Congestion Control paper fully described and characterized.…”

“…The system architecture of the solution presented in this publication extends the MR human-robot interface described in Section B of [10]. Specifically, this system implements all the functionalities available for that interface on screens, and extends it in the following aspects:…”

“…In [9], which focused on AR control of unmanned aerial vehicles (UAVs), it was emphasized that teleoperation hid the mapping between the operator's input and robot's dynamics, which can be learned only by experience, and presents a dangerous situation for untrained operators. While evaluating the MR human-robot interface in [10], the problems of predicting motion, spatial awareness in an unstructured fragile environment were addressed with preview and collision avoidance functionalities.…”