Ultrasound Mid-Air Tactile Feedback for Immersive Virtual Reality Interaction

Abstract

Ultrasound mid-air haptic (UMH) devices are promising for tactile feedback in virtual reality (VR), as they do not require users to be tethered to, hold, or wear any device. This approach is less cumbersome, easy to set up, can simplify tracking, and leaves the hands free for concurrent interactions. This chapter explores work conducted at CNRS-IRISA dealing with the challenges arising from the integration of UMH interfaces in immersive VR through three main axes. These are discussed in the wider context of the state-of-the-art on UMH for augmented and virtual reality, and illustrated through several VR use-cases. A first axis deals with device integration into the VR ecosystem. Interaction in immersive VR is based on the synergy between complex input devices allowing real-time tracking of the user and multimodal feedback devices delivering a coherent visual, auditory and haptic picture of a simulated virtual environment (VE). Using UMH in immersive VR therefore hinges on integrating UMH devices such that their operation does not interfere with other input and feedback devices. It is also critical to ensure that UMH feedback is adequately synchronised and co-located with respect to other stimuli, and delivered within a workspace that is compatible with that of VR interaction. Regarding this final point, we propose PUMAH, a robotic solution for increasing the usable workspace of UMH devices. The second and third axes, respectively, focus on stimulus perception and rendering of VE properties. Virtual object properties can be rendered in a variety of ways, through e.g. amplitude modulation (AM) or spatio-temporal modulation (STM), with many parameters (modulation frequency, spatial sampling, etc.) coming into play, raising questions about the limits of the design space. To tackle this challenge, we begin by conducting psychophysical experimentation to understand the usable ranges for stimulus parameters and understand the perceptual implications of stimulus design choices. We propose an open-source software framework intended to facilitate UMH stimulus design and perceptual evaluation. These results in turn serve as the basis for the design and evaluation of rendering schemes for VR. Using amplitude variations along a focal point path in STM, we investigate the possibility of rendering geometric details and in a second step, sensations of stiffness in VR.

An Interactive Tool for Simulating Mid-Air Ultrasound Tactons on the Skin

“…We selected 𝑓 𝐴𝑀 = 0 Hz to test pure STM rendering (𝑃𝑢𝑟𝑒𝑆𝑇 𝑀) and 𝑓 𝐴𝑀 > 60 Hz, as the power lines in our country introduces a constant 60 Hz measurement noise. We also used four combinations of size and drawing speed as (𝑑 in 𝑚𝑚, 𝑣 in 𝑚/𝑠): (0, 0), (10,6), (20,12), (30,18). We chose (0 𝑚𝑚, 0 𝑚/𝑠) to test pure AM rendering (𝑃𝑢𝑟𝑒𝐴𝑀), and maintained the same ratio of 𝑣 𝑑 for the other three combinations to have the same drawing frequency at 191 Hz.…”

“…This technology focuses acoustic waves into one or multiple focal points using a phased array of ultrasonic transducers, and modulates or moves these focal points to create a sense of touch [24]. Designers are exploring the parameter space of the mid-air ultrasound patterns (i.e., tactile icons or Tactons) to deliver information or emotion to users in various applications, including touchless public displays [18,31], automotive user interfaces [1,8], medical training simulations [13,14], and virtual reality environments [12,16,20,32].…”

Survey on Hand-Based Haptic Interaction for Virtual Reality

“…Howard et al [115] proposed a robotic solution to increase the usable workspace of ultrasound mid-air haptic devices, thus to enable the adequate synchronization and co-location of ultrasound tactile feedback with other stimuli for VR interaction. An open-source software framework [127] was also proposed for the stimulus design of the ultrasound tactile rendering and perceptual evaluation.…”

Designing Distinguishable Mid-Air Ultrasound Tactons with Temporal Parameters

“…To create different sensations, designers can modulate the parameters of the ultrasound pattern over time [41,68] or move the ultrasound focal point in space to render a shape on the user's skin [18,38]. The resulting mid-air ultrasound patterns, i.e., tactile icons or Tactons, can communicate information or emotion to users in various applications such as touchless interactions with public displays [37,66], automotive user interfaces [19], and virtual reality environments [23,27].…”