Communicating Robot Arm Motion Intent Through Mixed Reality Head-Mounted Displays

Abstract

Efficient motion intent communication is necessary for safe and collaborative work environments with collocated humans and robots. Humans efficiently communicate their motion intent to other humans through gestures, gaze, and social cues. However, robots often have difficulty efficiently communicating their motion intent to humans via these methods. Many existing methods for robot motion intent communication rely on 2D displays, which require the human to continually pause their work and check a visualization. We propose a mixed reality head-mounted display visualization of the proposed robot motion over the wearer's real-world view of the robot and its environment. To evaluate the effectiveness of this system against a 2D display visualization and against no visualization, we asked 32 participants to labeled different robot arm motions as either colliding or non-colliding with blocks on a table. We found a 16% increase in accuracy with a 62% decrease in the time it took to complete the task compared to the next best system. This demonstrates that a mixed-reality HMD allows a human to more quickly and accurately tell where the robot is going to move than the compared baselines.

Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

“…On the other hand, Virtual Reality teleoperation allows for a direct mapping of observations and actions between the teacher and the robot and does not suffer from the above correspondence issues [3], while also leveraging the natural manipulation instincts that the human teacher possesses. In a non-learning setting, VR teleoperation has been recently explored for controlling humanoid robots [40], [41], [42], for simulated dexterous manipulation [43], and for communicating motion intent [44]. Existing use cases of VR for learning policies have so far been limited to collecting waypoints of low-dimensional robot states [45], [46].…”

A Comparison of Visualisation Methods for Disambiguating Verbal Requests in Human-Robot Interaction

“…The application of mixed reality to human-robot interaction is an emerging field of research and shows promising results. For instance, Rosen et al [19] proposed a mixed reality framework to Figure 3: The architecture of the proposed system for visualising ambiguous fetching requests.…”

“…Similarly, because participants need to look away from the workspace and back at the monitor in MO, this will likely increase the number of errors. Our reasoning for establishing H3 and H5 is drawn from previous research showing that mixed reality applications can improve user experience [16,19]. H4 is argued for by reasoning that the augmented reality condition will enable participants to dedicate full attention to the workspace.…”

“…An alternative approach consists of employing visualisation techniques such as augmented [2] or mixed reality [8] to augment the scene with the robot's or human's intentions. While the first few works in this direction have started to appear [4,16,19], the effects of augmenting the workspace to disambiguate user verbal requests are still unknown.…”

Near-future perception system: Previewed Reality

“…In recent years, immersive VR display devices such as Oculus Rift (Oculus VR), augmented reality (AR) display devices such as HoloLens (Microsoft), or smart glasses with screens such as Epson Moverio, are beginning to be used in human-robot interaction [16][17][18], knowledge transfer [19], and human-robot collaborative manufacturing [20,21]. Most of VR display devices are immersivetype and need to attach cameras if a user wants to see a real scene.…”