Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing

Abstract

Persons with hearing loss use visual signals such as gestures and lip movement to interpret speech. While hearing aids and cochlear implants can improve sound recognition, they generally do not help the wearer localize sound necessary to leverage these visual cues. In this paper, we design and evaluate visualizations for spatially locating sound on a headmounted display (HMD). To investigate this design space, we developed eight high-level visual sound feedback dimensions. For each dimension, we created 3-12 example visualizations and evaluated these as a design probe with 24 deaf and hard of hearing participants (Study 1). We then implemented a real-time proof-of-concept HMD prototype and solicited feedback from 4 new participants (Study 2). Study 1 findings reaffirm past work on challenges faced by persons with hearing loss in group conversations, provide support for the general idea of sound awareness visualizations on HMDs, and reveal preferences for specific design options. Although preliminary, Study 2 further contextualizes the design probe and uncovers directions for future work.

Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies

“…Early work by Matthews et al [27] examined sound awareness needs across a variety of contexts (at home, at work, while mobile), and built and evaluated a proof-of-concept prototype to display office sounds on a computer monitor. More recent solutions have begun to investigate other form factors, including head-mounted displays for speech captioning [16,17,34], wrist-worn or smartwatch displays [20,29], and smartphone apps for general sound detection [3,30,39]. While formative studies and, in some cases, evaluations of these technologies have yielded useful insights, the studies tend to be qualitative and/or focused on a single device form factor and have not been designed to examine issues around social acceptability.…”

Exploring Sound Awareness in the Home for People who are Deaf or Hard of Hearing

“…Finally, wearable solutions for sound awareness have also emerged. For example, Jain et al [11] used a design probe method to explore sound visualizations on a headmounted display with 24 DHH participants. Mielke et al [14] conducted a Wizard of Oz exploration of a smartwatch-based app with six DHH participants.…”

An Evaluation of Radar Metaphors for Providing Directional Stimuli Using Non-Verbal Sound

“…Advances in AR headsets create opportunities to present the environment to users in new and novel ways. Much research in this area focuses on using headset displays to visualize parts of the environment not normally seen, such as the visualization of sound fields [10] or visualizations for spatially locating sound [11]. Less research exists on presenting local environmental information (such as spatial mapping data) using audio.…”