Design and Evaluation of AR-Assisted End-User Robot Path Planning Strategies

Abstract

Nowadays, robots play an increasingly important role but still usually have to be programmed by highly skilled professionals. Therefore, end-user solutions supporting users in solving (simple) robot programming tasks without expert knowledge are a promising research field. A possibility for these solutions is the inclusion of Augmented Reality (AR) to enable users to work in the robot space, reducing the amount of mentally taxing coordinate space conversions. Approaches for this mainly rely on the waypoint-based robot path programming strategy. To explore an alternative solution, we propose an AR-assisted approach with different possible path-planning strategies, such as drawing paths or selecting single waypoints directly in the real world. This enables end-users without a programming background to program paths for a wheeled mobile robot. They can also see and edit their programmed paths in a Blockly-like representation. Furthermore, we offer AR in-place program simulation and direct building of finished programs to the real robot. We evaluated our approach regarding usability compared to existing non-AR end-user software as well as comparing the two different path-planning strategies to each other. The evaluation showed that our approach is more usable and faster than the conventional method, while the differences between the path-planning methods are more nuanced and show qualities in both versions. CCS CONCEPTS• Human-centered computing → Human computer interaction; Visualization; • Software and its engineering → Software notations and tools.

Keep the Human in the Loop: Arguments for Human Assistance in the Synthesis of Simulation Data for Robot Training

“…Mixed Reality (MR) applications are increasingly being applied to human-robotics interaction (HRI), mostly for teleoperation, simulation, or explaining robot actions [27]. These applications enable humans without a robotics background to use their knowledge of the application environment to expand the robot's operating range [28], provide a path [29], and validate automated systems using live visualization of simulation environments with human actions [30]. While these approaches address the interaction between robots and humans, our concept explores how application experts can influence robot training.…”

“…As application experts are familiar with the robot's operating environment [29,42], they can detect missing information and identify errors (e.g., missing objects) by reviewing the initial recordings; thereby, they can also assess scan quality. For instance, they may assist in filtering the recordings by selecting high-quality images or deleting faulty ones, thus improving the overall acquisition set.…”

“…Humans excel at object recognition [38], which makes them valuable for improving the computed virtual representations of recordings using their scene understanding [29,42]. To enhance the recorded virtual scans, they could specify data points, extend planar surfaces, insert single or clusters of data, draw missing objects, or add objects from collections.…”

“…There are multiple technologies available through which application experts can be included in the robot training process, such as desktop solutions and MR technologies [25]. However, MR technologies, such as Augmented Reality (AR) and Virtual Reality (VR), offer great potential because they facilitate intuitive interaction with visual representations in three-dimensional space [8,29]. Thus, we choose to use VR to enable remote operation, allowing persons not physically present to interact with the system.…”

Creating and manipulating 3D paths with mixed reality spatial interfaces

“…In a small pilot study comparing the proposed AR interface with a kinesthetic gamepad interface, researchers found that while the AR interface took less time to learn, it introduced a mental workload nearly five times higher than the gamepad method during path creation. Krings et al (2022) presented an AR-assisted path creation system for ground-constrained robot paths that tested both waypoint-based planning and path drawing against Mindstorms 1 programming. In an initial evaluation with 18 participants, AR path creation outperformed the computer-based Mindstorms approach in usability and efficiency, likely because it eliminated the need for trial and error when programming positions.…”