Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

Abstract

Remote robotic interventions and maintenance tasks are frequently required in hazardous environments. Particularly, missions with a redundant mobile manipulator in the world's most complex machine, the CERN Large Hadron Collider, are performed in a sensitive underground environment with radioactive or electromagnetic hazards, bringing further challenges in safety and reliability. The mission's success depends on the robot's hardware and software, and when the tasks become too unpredictable to execute autonomously, the operators need to make critical decisions. However, in most current humanmachine systems, the state of the human is neglected. In this context, a novel 3D Mixed Reality (MR) human-robot interface with the Operator Monitoring System (OMS) was developed to advance safety and task efficiency with improved spatial awareness, advanced manipulator control, and collision avoidance. However, new techniques could increase the system's sophistication and add to the operator's workload and stress. Therefore, for operational validation, the 3D MR interface had to be compared with an operational 2D interface, which has been used in hundreds of interventions. With the 3D MR interface, the execution of precise approach tasks was faster, with no increased workload or physiological response. The new 3D MR techniques improved the teleoperation quality and safety while maintaining similar effects on the operator. The OMS worked jointly with the interface and performed well with operators with varied teleoperation backgrounds facing a stressful real telerobotic scenario in the LHC. The paper contributes to the methodology for human-centred interface evaluation incorporating the user's physiological state: heart rate, respiration rate and skin electrodermal activity, and combines it with the NASA TLX assessment method, questionnaires, and task execution time. It provides novel approaches to operator state identification, the GUI-OMS software architecture, and the evaluation of the 3D MR techniques. The solutions can be practically applied in mission-critical applications, such as telesurgery, space robotics, uncrewed transport vehicles and semi-autonomous machinery.

Enhancing Robot Behavior with EEG, Reinforcement Learning and Beyond: A Review of Techniques in Collaborative Robotics

“…In the field of augmented reality and collaborative robotics, a novel solution was proposed, involving a human-robot collaboration method using augmented reality to enhance construction waste sorting, aiming to improve both efficiency and safety [70]. Mixed reality is also becoming more popular due to some recently released products, some interesting research ensure the possibility of applying mixed reality in robotic environments for enhancing the interactions between humans and robots [71].…”

Using skeletal position to estimate human error rates in telemanipulator operators

“…Machine learning techniques were then applied to estimate operator safety factors based on bio-mechanical features, offering the potential for real-time safety warnings in shared control models. Szczurek et al (2023) state that the success of teleoperations hinges on three key aspects: Operators must effectively achieve mission goals, adapt to unexpected challenges while minimising risks to human life and equipment, and remain aware of their changing capabilities throughout the task. They further assert that these factors are closely tied to the operator's emotional and physiological state, which in turn impacts the operational success rate.…”