A user study to compare two conversational assistants designed for people with hearing impairments

Abstract

Participating in conversations can be difficult for people with hearing loss, especially in acoustically challenging environments. We studied the preferences the hearing impaired have for a personal conversation assistant based on automatic speech recognition (ASR) technology. We created two prototypes which were evaluated by hearing impaired test users. This paper qualitatively compares the two based on the feedback obtained from the tests. The first prototype was a proof-of-concept system running real-time ASR on a laptop. The second prototype was developed for a mobile device with the recognizer running on a separate server. In the mobile device, augmented reality (AR) was used to help the hearing impaired observe gestures and lip movements of the speaker simultaneously with the transcriptions. Several testers found the systems useful enough to use in their daily lives, with majority preferring the mobile AR version. The biggest concern of the testers was the accuracy of the transcriptions and the lack of speaker identification.

You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarification

“…Research notes that people with partial hearing loss really struggle to follow a conversation in a noisy environment (like a public space), so screens have been successfully used to live transcribe the ongoing conversation (Lukkarila et al, 2017;Virkkunen et al, 2019), improving their feeling of inclusion, and systems have been created using sign language (Mande et al, 2021;Yin et al, 2021;Glasser et al, 2022;Inan et al, 2022). Prototype EVAs have been developed for people with speech impairments (Hawley et al, 2007(Hawley et al, , 2012Derboven et al, 2014;Jamal et al, 2017), but Google is currently pioneering this front with three projects.…”

“…Minimally verbal children with autism learn to use novel vocabulary after long-term EVA use (Kasari et al, 2014 ), and similar work effectively used interactive social robots for autism therapy (Cabibihan et al, 2013 ; Pennisi et al, 2016 ), but no research focused on improving the system's speech processing and understanding . Research notes that people with partial hearing loss really struggle to follow a conversation in a noisy environment (like a public space), so screens have been successfully used to live transcribe the ongoing conversation (Lukkarila et al, 2017 ; Virkkunen et al, 2019 ), improving their feeling of inclusion, and systems have been created using sign language (Mande et al, 2021 ; Yin et al, 2021 ; Glasser et al, 2022 ; Inan et al, 2022 ). Prototype EVAs have been developed for people with speech impairments (Hawley et al, 2007 , 2012 ; Derboven et al, 2014 ; Jamal et al, 2017 ), but Google is currently pioneering this front with three projects.…”