Negotiation-based Human-Robot Collaboration via Augmented Reality

Virtual, Augmented, and Mixed Reality for Human-Robot Interaction: A Survey and Virtual Design Element Taxonomy

“…The rise of new VAM-HRI research has demonstrated the potential for virtual reality (VR), augmented reality (AR), and mixed reality (MR) across a number of application domains that are of significant historical interest to the field of HRI, including education [15,27], social support [32,67], and task-based collaboration [14,22], while illustrating how MR and VR can be used as new tools in a HRI researcher's toolkit for robot design [12], human-subject experimentation [93,95], and robot programming and debugging [35].…”

“…Robot Location VDEs convey where a robot is in the environment, e.g., as an occluded robot's outline with a Spatial Visualization Virtual Robot or a virtual indicator on the other side of a wall or door, as a top-down radar-like display showing user and robot locations, or as an off-screen indicator to direct a user's attention to a robot outside the current field-of-view [14,90] (see Figure 5-D).…”

“…These visualizations enable users to easily reference entities in spoken commands without the need for referring expressions such as enabling users to instruct robots through commands such as "pick up cube B" [8,76] (see Figure 4-H). Additionally, these VDEs allow robots to label points of importance in the environment such as a room's primary access point [14] (see Figure 5-D) Entity Attributes convey information a robot knows about an entity's characteristics, such as whether an entity is heavy, delicate, or dangerous; information known about the entity's affordances; the current state of an entity (e.g., an entity that's too hot, in a dangerous location, still drying, sleeping, charging its battery, etc. ); or an entity's geometry and shape (e.g., optimal grasp points or surface normals) [13] (see Figure 5-A).…”

“…5. A: Entity Attributes and Alteration Previews [13], B: Entity Appearances [26], C: Command Options [67], D: Visualization Robots, Robot Locations, and Entity Labels [14], E: Task Instructions [26], F: Virtual Controllers [46], G: Simulated Objects [9], H: Spatial Previews and Object Digital Twins [23].…”

HAVEN: A Unity-based Virtual Robot Environment to Showcase HRI-based Augmented Reality

“…The system consists of different visualizations that represent sensory and cognitive data that is rendered on the user interface. Other studies utilize AR interface to support human-multirobot collaboration, where the robots collaborate with both the robot and human teammates (Chandan, Li, and Zhang, 2019).…”