Towards the Design of Effective Haptic and Audio Displays for Augmented Reality and Mixed Reality Applications

Abstract

Augmented Reality (AR) and Mixed Reality (MR) technology has potential for supporting mobile applications. However, nonvisual interaction modalities are undervalued and underused in AR/MR applications. Visual displays can be ineffective or inappropriate in some situations such as walking or driving. Meanwhile, nonvisual modalities are becoming increasingly important in mobile user experiences. In this paper, we report two studies investigating nonvisual interaction modalities such as audio and haptic displays with mobile AR/MR applications. In the first study, we investigate a range of design factors for haptic and audio displays, including rhythm, amplitude, and their combination in representing tourism information to users with a mobile phone. The results show a main effect for Interaction modality, with identification rates highest for information represented in a combined Haptic-Audio display. In the second study, we investigate target location tasks in 3D space using spatial audio feedback and a head-mounted display. We evaluate several design factors including audio feedback device, volume, rhythm, and the target’s horizontal and vertical position. Results show that the vertical positions are very difficult to locate, and overall our participants prefer audio cues with loud volume and fast rhythm. Finally, we propose practical audio and haptic display design guidelines for AR/MR applications.

A Perspective Review on Integrating VR/AR with Haptics into STEM Education for Multi-Sensory Learning

“…Less common forms of display are: tactile, audio and olfactory. Notably, audio displays are still much more common than tactile and olfactory displays, as there are already consumer devices specifically aimed at audio AR [61]. Input devices are often considered as a separate category, ranging from keyboards to voice inputs.…”

Audio Legends: Investigating Sonic Interaction in an Augmented Reality Audio Game

“…Audio interaction can help users navigate to locations without distracting them from their environment [33] and regardless of their musical training [34]. Another study suggests that the visual display bears some weaknesses that the audio modality can overcome, such as limited screen space, overload of information, vulnerability to sunlight, and the necessity for constant attention [35]. Furthermore, many experiments argue that augmentation through data sonification and sonic interaction design techniques can significantly enhance players' immersion level in the augmented environment [29,36] and increase their emotional engagement with the virtual world [37,38].…”

“…Nevertheless, the notion that ARA is still under-researched and needs to be systematically studied is widely supported in the literature. Sound design is only given a marginal role in the development of location based games [36], and there is a lack of design guidelines for non-visual display modes of AR applications [35]. Sound appears to be an unfamiliar game mechanic to sighted players [40].…”