ARROCH: Augmented Reality for Robots Collaborating with a Human

Chandan, Kishan; Kudalkar, Vidisha; Li, Xiang; Zhang, Shiqi

doi:10.1109/icra48506.2021.9561144

Cited by 21 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been multiple recent studies evaluating wearable AR technology for human-robot interaction while sharing a virtual or physical workspace of a single operator with a single mobile robot [9] [23] [24] [25] [26] or a fixed manipulator [27] [28]. Some studies were done for a single operator controlling multiple robots [24] [29] [30]. Table 1 presents an analysis of selected MR human-robot interfaces relevant to the research and development of the work presented in this paper.…”

Section: B State-of-the-art Of Wearable Ar Technology For Multi-user ...mentioning

confidence: 99%

Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

et al. 2023

View full text Add to dashboard Cite

In hazardous environments, where conditions present risks for humans, the maintenance and interventions are often done with teleoperated remote systems or mobile robotic manipulators to avoid human exposure to dangers. The increasing need for safe and efficient teleoperation requires advanced environmental awareness and collision avoidance. The up-to-date screen-based 2D or 3D interfaces do not fully allow the operator to immerse in the controlled scenario. This problem can be addressed with the emerging Mixed Reality (MR) technologies with Head-Mounted Devices (HMDs) that offer stereoscopic immersion and interaction with virtual objects. Such human-robot interfaces have not yet been demonstrated in telerobotic interventions in particle physics accelerators. Moreover, robotic operations often require a few experts to collaborate, which increases the system complexity and requires sharing a multi-user Augmented Reality (AR) workspace. The multi-user telerobotics with shared control in the AR has not yet been approached in the industrial state-of-the-art. In this work, the developed MR human-robot interface using the AR HMD is presented. The interface adapts to the constrained wireless networks in particle accelerator facilities and provides reliable high-precision interaction and specialized visualization. The multimodal operation uses hands, eyes and user motion tracking, and voice recognition for control, as well as offers video, 3D point cloud and audio feedback from the robot. Multiple experts can collaborate in the AR workspace locally or remotely, share the robot's control and monitor robotic teleoperation. Ten (10) operators tested the interface in intervention scenarios in the European Organization for Nuclear Research (CERN) with complete network characterization and measurements to conclude if operational requirements were met and if the network architecture could support single and multi-user communication load.The interface system has proved to be operationally ready at the Technical Readiness Level (TRL) 8 -and was validated through successful tests and demonstration in single and multi-user missions. Some areas of system limitations and further work were identified, such as optimising the network architecture for multi-user scenarios or high-level interface actions applying automatic interaction strategies with the robot depending on network conditions.

show abstract

Section: B State-of-the-art Of Wearable Ar Technology For Multi-user ...mentioning

confidence: 99%

Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Ostatin [20] and Togias [24] both use VR to plan the robotic arm's trajectory. Others, like Chandan et al [2] use AR to visualize the states and intentions of the robots. Xu et al [31] used VR to steer the robot by moving its end effector in the VR space.…”

Section: Related Workmentioning

confidence: 99%

“…Our system then uses this information to correctly position and add objects to the scene in VR. 2 Any other object detection architectures could be used as well. The VR environment is then built from the output of the perception module.…”

Section: Camera Views and Perception Modulementioning

confidence: 99%

What you see is (not) what you get: A VR Framework for Correcting Robot Errors

Wozniak,

Stower,

Jensfelt

et al. 2023

Preprint

View full text Add to dashboard Cite

Many solutions tailored for intuitive visualization or teleoperation of virtual, augmented and mixed (VAM) reality systems are not robust to robot failures, such as the inability to detect and recognize objects in the environment or planning unsafe trajectories. In this paper, we present a novel virtual reality (VR) framework where users can (i) recognize when the robot has failed to detect a realworld object, (ii) correct the error in VR, (iii) modify proposed object trajectories and, (iv) implement behaviors on a real-world robot. Finally, we propose a user study aimed at testing the efficacy of our framework. Project materials can be found in the OSF repository 1 . CCS CONCEPTS• Human-centered computing → Virtual reality.

show abstract

“…Prior work in this domain has explored the use of augmented reality (AR) for more efficient human-robot interaction and visualization of the robot's state and intent. However, these works have relied on colocalization through detection of landmark objects [8], or the use of fiducial markers mounted on the robot [9]. While these approaches for colocalization are sufficient to perform AR visualization, they do not provide further spatial understanding for the AR device Fig.…”

Section: A Colocalizing the Human And Robotmentioning

confidence: 99%

Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together

Delmerico,

Poranne,

Bogo

et al. 2022

Preprint

View full text Add to dashboard Cite

Spatial computing-the ability of devices to be aware of their surroundings and to represent this digitally-offers novel capabilities in human-robot interaction. In particular, the combination of spatial computing and egocentric sensing on mixed reality devices enables them to capture and understand human actions and translate these to actions with spatial meaning, which offers exciting new possibilities for collaboration between humans and robots. This paper presents several human-robot systems that utilize these capabilities to enable novel robot use cases: mission planning for inspection, gesture-based control, and immersive teleoperation. These works demonstrate the power of mixed reality as a tool for human-robot interaction, and the potential of spatial computing and mixed reality to drive the future of human-robot interaction.

show abstract

ARROCH: Augmented Reality for Robots Collaborating with a Human

Cited by 21 publications

References 26 publications

Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

What you see is (not) what you get: A VR Framework for Correcting Robot Errors

Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together

Contact Info

Product

Resources

About