Multimodal Mixed Reality Impact on a Hand Guiding Task with a Holographic Cobot

Abstract

In the context of industrial production, a worker that wants to program a robot using the hand-guidance technique needs that the robot is available to be programmed and not in operation. This means that production with that robot is stopped during that time. A way around this constraint is to perform the same manual guidance steps on a holographic representation of the digital twin of the robot, using augmented reality technologies. However, this presents the limitation of a lack of tangibility of the visual holograms that the user tries to grab. We present an interface in which some of the tangibility is provided through ultrasound-based mid-air haptics actuation. We report a user study that evaluates the impact that the presence of such haptic feedback may have on a pick-and-place task of the wrist of a holographic robot arm.

User Experience and Mid-Air Haptics: Applications, Methods, and Challenges

“…Similar observations were made by Pinto et al who explored pick-andplace tasks within a mixed reality robotic teleportation environment [60]. In their implementation, the authors were looking to teach a robotic arm how to perform such tasks without using any kinematic or programming languages, but instead through human hand guidance, i.e., mimicking of user movements.…”

“…While many of these key targets relate to the overall touchless system's performance, and therefore depend on a complex set of sub-components and their interactions, several UX studies of ultrasound mid-air haptics have reported relevant benefits thus motivating their use in touchless medical prototype systems with haptic enhanced gesture input. For example, studies have demonstrated how mid-air haptics can help localize and interact with floating widgets [49,73], enhance grasping of virtual objects in VR [61] and AR [74], improve the sense of agency [22], enhance perceived physicality of holograms [60], and improve usability and aesthetic appeal [38].…”

Shaping the role of the digital twins for human‐robot dyad: Connotations, scenarios, and future perspectives

“…Proposition 3: Digital twins are handy tools to control the robot using gestures and voice control remotely Digital Twins of human-robot systems could integrate voice and gesture control using techniques like Natural Language Processing and gesture recognition to control robots in an interactive environment remotely [42]. Similarly, DTs can allow the physical robots to be re-programed based on their imitation of a procedure demonstrated to its digital copy [49].…”

“…Garg et al [34] used another communication protocol called Socket messaging based on Client-Server architecture. Similarly, some applications use classical communication styles, such as Bluetooth [52] and wireless communication [49].…”

“…Similarly, Liang et al [43] use the iso camera to find the robot control policy of the construction task with the Trust Region Policy Gradient method. Other emerging hardware components in DT-HRI literature are Haptics [33,49,57] and Radio Frequency Identification [47]. However, they are scarcely utilised in the current literature but could shape an interesting direction for future research.…”