Deaf and Hard-of-Hearing Perspectives on Imperfect Automatic Speech Recognition for Captioning One-on-One Meetings

Berke, Larwan; Caulfield, Christopher; Huenerfauth, Matt

doi:10.1145/3132525.3132541

Cited by 42 publications

(11 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To sum up, the perceived usefulness of the compressed subtitles is mixed with a tendency towards full subtitles because the participants are used to it (as stated in their final feedback). This is in line with the findings of Berke et al [37], who also observe that people tend to reject unknown subtitle designs.…”

Section: Resultssupporting

confidence: 92%

Automatic Compression of Subtitles with Neural Networks and its Effect on User Experience

Angerbauer¹,

Adel²,

Vu³

2019

Interspeech 2019

View full text Add to dashboard Cite

Understanding spoken language can be impeded through factors like noisy environments, hearing impairments or lack of proficiency. Subtitles can help in those cases. However, for fast speech or limited screen size, it might be advantageous to compress the subtitles to their most relevant content. Therefore, we address automatic sentence compression in this paper. We propose a neural network model based on an encoder-decoder approach with the possibility of integrating the desired compression ratio. Using this model, we conduct a user study to investigate the effects of compressed subtitles on user experience. Our results show that compressed subtitles can suffice for comprehension but may pose additional cognitive load.

show abstract

Section: Resultssupporting

confidence: 92%

Automatic Compression of Subtitles with Neural Networks and its Effect on User Experience

Angerbauer¹,

Adel²,

Vu³

2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…Users are engaged in an attentiondemanding task of viewing a video with multiple sources of visual information in parallel to the caption-text stream, and if the choice of highlighting style or the frequency with which words are highlighted is suboptimal, then such visual decoration of the text could be distracting. This speculation is supported by prior research that has investigated the effect of different visual markup of caption texts in another context: to convey the confidence scores of a captioning service (e.g., ASR system) as to the accuracy of its caption output [6]. In a study with 107 DHH users, Berke et al discovered that although participants were receptive to the idea of having visual indicators of the confidence of an automatic caption system, they were concerned about distraction from changes in text appearance.…”

Section: Visual Markup Of Text In Captionsmentioning

confidence: 91%

“…Such highlighting in captions may require special consideration: Unlike text documents, captions are dynamic, with shorter text segments, which are usually shown in 1 or 2 lines, for 2 to 4 seconds [28]. Moreover, users are known to be sensitive to caption display parameters such as speed, font size, or decorations: Several researchers have measured the influence of such visual parameters of caption appearance on the readability of captions for DHH users [6,28,43].…”

Section: Introductionmentioning

confidence: 99%

Evaluating the Benefit of Highlighting Key Words in Captions for People who are Deaf or Hard of Hearing

Kafle

Yeung

Huenerfauth

2019

The 21st International ACM SIGACCESS Conference on Computers and Accessibility

Self Cite

View full text Add to dashboard Cite

Recent research has investigated automatic methods for identifying how important each word in a text is for the overall message, in the context of people who are Deaf and Hard of Hearing (DHH) viewing video with captions. We examine whether DHH users report benefits from visual highlighting of important words in video captions. In formative interview and prototype studies, users indicated a preference for underlining of 5%-15% of words in a caption text to indicate that they are important, and they expressed an interest for such text markup in the context of educational lecture videos. In a subsequent user study, 30 DHH participants viewed lecture videos in two forms: with and without such visual markup. Users indicated that the videos with captions containing highlighted words were easier to read and follow, with lower perceived task-load ratings, compared to the videos without highlighting. This study motivates future research on caption highlighting in online educational videos, and it provides a foundation for how to evaluate the efficacy of such systems with users.

show abstract

“…they may fail on unexpected cases or fail in ways that are unlike how humans fail. In our research on using ASR to automatically generate captions for DHH users during live conversations with hearing colleagues, we have found that users are still interested in having access to such technology, even if it is not yet perfect [5]. However, they would like to have more information to help them, as an end-user, determine when they should trust the output of these systems.…”

Section: Lack Of Interpretabilitymentioning

confidence: 99%

Artificial intelligence fairness in the context of accessibility research on intelligent systems for people who are deaf or hard of hearing

Kafle

Glasser

Al-khazraji

et al. 2020

SIGACCESS Access. Comput.

Self Cite

View full text Add to dashboard Cite

We discuss issues of Artificial Intelligence (AI) fairness for people with disabilities, with examples drawn from our research on HCI for AI-based systems for people who are Deaf or Hard of Hearing (DHH). In particular, we discuss the need for inclusion of data from people with disabilities in training sets, the lack of interpretability of AI systems, ethical responsibilities of access technology researchers and companies, the need for appropriate evaluation metrics for AI-based access technologies (to determine if they are ready to be deployed and if they can be trusted by users), and the ways in which AI systems influence human behavior and influence the set of abilities needed by users to successfully interact with computing systems.

show abstract

Deaf and Hard-of-Hearing Perspectives on Imperfect Automatic Speech Recognition for Captioning One-on-One Meetings

Cited by 42 publications

References 36 publications

Automatic Compression of Subtitles with Neural Networks and its Effect on User Experience

Automatic Compression of Subtitles with Neural Networks and its Effect on User Experience

Evaluating the Benefit of Highlighting Key Words in Captions for People who are Deaf or Hard of Hearing

Artificial intelligence fairness in the context of accessibility research on intelligent systems for people who are deaf or hard of hearing

Contact Info

Product

Resources

About