Deaf, Hard of Hearing, and Hearing Perspectives on Using Automatic Speech Recognition in Conversation

Glasser, Abraham; Kushalnagar, Kesavan; Kushalnagar, Raja

doi:10.1145/3132525.3134781

Cited by 22 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HMDs have also been used to display pre-recorded captions for moviegoers [40] and pre-recorded sign language interpretations in educational environments [11,19,33]. Offthe-shelf text-to-speech functionality on smartphones can also support communication between deaf and hearing persons, although important challenges with speech recognition accuracy exist, particularly with recognizing deaf speech [8]. While this work illustrates the increasing interest in mobile and wearable sound awareness, studies to date have only briefly [17,39] compared form factors and design options, which our survey does in more depth.…”

Section: Visual and Haptic Sound Awareness Approachesmentioning

confidence: 99%

Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies

Findlater

Chinh

Jain

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

To investigate preferences for mobile and wearable sound awareness systems, we conducted an online survey with 201 DHH participants. The survey explores how demographic factors affect perceptions of sound awareness technologies, gauges interest in specific sounds and sound characteristics, solicits reactions to three design scenarios (smartphone, smartwatch, head-mounted display) and two output modalities (visual, haptic), and probes issues related to social context of use. While most participants were highly interested in being aware of sounds, this interest was modulated by communication preference-that is, for sign or oral communication or both. Almost all participants wanted both visual and haptic feedback and 75% preferred to have that feedback on separate devices (e.g., haptic on smartwatch, visual on head-mounted display). Other findings related to sound type, full captions vs. keywords, sound filtering, notification styles, and social context provide direct guidance for the design of future mobile and wearable sound awareness systems.

show abstract

Section: Visual and Haptic Sound Awareness Approachesmentioning

confidence: 99%

Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies

Findlater

Chinh

Jain

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…The usability, processing speed, effect on speechreading, and readability of the transcript were not evaluated. Other researchers looked into requirements for speed and user interface and concluded that those are important factors to improve usability ( 39 ). We expect that an increasing number of ASR apps will adhere to accessibility guidelines to improve usability for the elderly and people with disabilities as promoted by the Web Accessibility Initiative ( 40 ).…”

Section: Discussionmentioning

confidence: 99%

Preliminary Evaluation of Automated Speech Recognition Apps for the Hearing Impaired and Deaf

Pragt¹,

Hengel²,

Grob³

et al. 2022

Front. Digit. Health

View full text Add to dashboard Cite

ObjectiveAutomated speech recognition (ASR) systems have become increasingly sophisticated, accurate, and deployable on many digital devices, including on a smartphone. This pilot study aims to examine the speech recognition performance of ASR apps using audiological speech tests. In addition, we compare ASR speech recognition performance to normal hearing and hearing impaired listeners and evaluate if standard clinical audiological tests are a meaningful and quick measure of the performance of ASR apps.MethodsFour apps have been tested on a smartphone, respectively AVA, Earfy, Live Transcribe, and Speechy. The Dutch audiological speech tests performed were speech audiometry in quiet (Dutch CNC-test), Digits-in-Noise (DIN)-test with steady-state speech-shaped noise, sentences in quiet and in averaged long-term speech-shaped spectrum noise (Plomp-test). For comparison, the app's ability to transcribe a spoken dialogue (Dutch and English) was tested.ResultsAll apps scored at least 50% phonemes correct on the Dutch CNC-test for a conversational speech intensity level (65 dB SPL) and achieved 90–100% phoneme recognition at higher intensity levels. On the DIN-test, AVA and Live Transcribe had the lowest (best) signal-to-noise ratio +8 dB. The lowest signal-to-noise measured with the Plomp-test was +8 to 9 dB for Earfy (Android) and Live Transcribe (Android). Overall, the word error rate for the dialogue in English (19–34%) was lower (better) than for the Dutch dialogue (25–66%).ConclusionThe performance of the apps was limited on audiological tests that provide little linguistic context or use low signal to noise levels. For Dutch audiological speech tests in quiet, ASR apps performed similarly to a person with a moderate hearing loss. In noise, the ASR apps performed more poorly than most profoundly deaf people using a hearing aid or cochlear implant. Adding new performance metrics including the semantic difference as a function of SNR and reverberation time could help to monitor and further improve ASR performance.

show abstract

“…However, the variability and complexity of speech often cause issues regarding recognition accuracy, caption latency, and context formalization [2,16,26,28,35,54,60]. Furthermore, background noise, multi-talker speech, human accent, and disfluent speech may further downgrade the quality of automatic captions [3,18,54]. To make automatic captions work better, prior work explored the approaches such as removing the noise from the environment and changing the appearances of the automatic captions to convey ASR confidence [4] (e.g., alternating the font size [56], font color [60], and underlining [70]).…”

Section: Existing Captioning Methodsmentioning

confidence: 99%

“…We then ended up filtering 59 videos and created the final video dataset with 189 videos (V1 -V189). Among 189 videos in our dataset, most videos were uploaded in 2016 (68), while others were uploaded in 2019 (31), 2017 (24), 2020 (23), 2018 (18). The average length of videos was 354 seconds (ranging from 27 seconds to 1157 seconds).…”

Section: Youtube Video Analysis-data Collectionmentioning

confidence: 99%

See 1 more Smart Citation

An Exploration of Captioning Practices and Challenges of Individual Content Creators on YouTube for People with Hearing Impairments

Li,

Lu,

et al. 2022

Preprint

View full text Add to dashboard Cite

Fig. 1. YouTubers' and Deaf and Hard-of-Hearing audiences' complaints and campaigns on asking for better captions of videos created by individual content creators on video-sharing platforms (e.g., #NoMoreCraptions). Deaf and Hard-of-Hearing (DHH) audiences have long complained about caption qualities for many online videos created by individual content creators on video-sharing platforms (e.g., YouTube). However, there lack explorations of practices, challenges, and perceptions of online video captions from the perspectives of both individual content creators and DHH audiences. In this work, we first explore DHH audiences' feedback on and reactions to YouTube video captions through interviews with 13 DHH individuals, and uncover DHH audiences' experiences, challenges, and perceptions on watching videos created by individual content creators (e.g., manually added caption tags could create additional confidence and trust in caption qualities for DHH audiences). We then discover individual content creators' practices, challenges, and perceptions on captioning their videos (e.g., back-captioning problems) by conducting a YouTube video analysis with 189 captioning-related YouTube videos, followed by a survey with 62 individual content creators. Overall, our findings provide an in-depth understanding of captions generated by individual content creators and bridge the knowledge gap mutually between content creators and DHH audiences on captions. CCS Concepts: • Human-centered computing → Empirical studies in accessibility.

show abstract

Deaf, Hard of Hearing, and Hearing Perspectives on Using Automatic Speech Recognition in Conversation

Cited by 22 publications

References 18 publications

Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies

Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies

Preliminary Evaluation of Automated Speech Recognition Apps for the Hearing Impaired and Deaf

An Exploration of Captioning Practices and Challenges of Individual Content Creators on YouTube for People with Hearing Impairments

Contact Info

Product

Resources

About