Preferred Appearance of Captions Generated by Automatic Speech Recognition for Deaf and Hard-of-Hearing Viewers

Berke, Larwan; Albusays, Khaled; Seita, Matthew; Huenerfauth, Matt

doi:10.1145/3290607.3312921

Cited by 35 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Human efforts are needed to correct those errors however video content creators were frequently unwilling or unable to do so [40,47]. Beyond the caption's accuracy, captions' appearance is also important to understanding [11]. Low-quality captions for online learning videos created great challenges for DHH learners to understand the learning content when studying remotely from school through video lectures during the COVID-19 period, and aggravated mental health issues [5].…”

Section: Related Work 21 Challenges Faced By Dhh Students In Video-ba...mentioning

confidence: 99%

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Chen,

Cheng,

Situ

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Previous research underscored the potential of danmaku-a textbased commenting feature on videos-in engaging hearing audiences. Yet, for many Deaf and hard-of-hearing (DHH) individuals, American Sign Language (ASL) takes precedence over English. To improve inclusivity, we introduce "Signmaku, " a new commenting mechanism that uses ASL, serving as a sign language counterpart to danmaku. Through a need-finding study (N=12) and a withinsubject experiment (N=20), we evaluated three design styles: real human faces, cartoon-like figures, and robotic representations. The results showed that cartoon-like signmaku not only entertained but also encouraged participants to create and share ASL comments, with fewer privacy concerns compared to the other designs. Conversely, the robotic representations faced challenges in accurately depicting hand movements and facial expressions, resulting in higher cognitive demands on users. Signmaku featuring real human faces elicited the lowest cognitive load and was the most comprehensible among all three types. Our findings offered novel design implications for leveraging generative AI to create signmaku comments, enriching co-learning experiences for DHH individuals.

show abstract

Section: Related Work 21 Challenges Faced By Dhh Students In Video-ba...mentioning

confidence: 99%

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Chen,

Cheng,

Situ

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Captions allow meeting participants to view text of spoken comments in real time. A variety of studies have documented the success of captions to provide more inclusionary communication for those who have hearing limitations (Berke et al, 2019;Kafle et al, 2019;McCarron, 2021) and for second-language speakers (Fleischmann et al, 2021).…”

Section: Smart Meeting Toolsmentioning

confidence: 99%

Acceptance of AI-Based Meeting Tools: Psychological Safety as a Foundation for Smart Collaboration

Cardon,

Fleischmann,

Carradini

et al. 2023

Preprint

View full text Add to dashboard Cite

Smart (AI-based) meeting tools are increasingly available for professionals to use during and after online meetings. The technological affordances of these tools hold promise for better team communication and collaboration. This study emerged from a simple premise: psychological safety influences key factors in technology acceptance, including performance expectancy and effort expectancy. Three surveys involving 509 frequent users of online work meetings were conducted to explore attitudes toward three forms of meeting tools: captions, recorded meetings, and AI meeting assistants. Results showed that psychological safety influences performance expectancy and effort expectancy. The study concludes with recommendations for future research and practice in terms of the fundamental importance of psychological safety for smart collaboration.

show abstract

“…The caption design has a significant effect on readability. Previous studies have investigated the captionappearance preferences [15] and designs based on the reliability of speech recognition [16]. The proposed system provides an interface that allows the user to change the caption-design parameters such as character size, character color, character transparency, and font.…”

Section: Caption Designmentioning

confidence: 99%

See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People

Yamamoto

Suzuki

Shitara

et al. 2021

Preprint

View full text Add to dashboard Cite

Real-time captioning is a useful technique for deaf and hard-of-hearing (DHH) people to talk to hearing people. With the improvement in device performance and the accuracy of automatic speech recognition (ASR), real-time captioning is becoming an important tool for helping DHH people in their daily lives. To realize higher-quality communication and overcome the limitations of mobile and augmented-reality devices, real-time captioning that can be used comfortably while maintaining nonverbal communication and preventing incorrect recognition is required. Therefore, we propose a real-time captioning system that uses a transparent display. In this system, the captions are presented on both sides of the display to address the problem of incorrect ASR, and the highly transparent display makes it possible to see both the body language and the captions.

show abstract

Preferred Appearance of Captions Generated by Automatic Speech Recognition for Deaf and Hard-of-Hearing Viewers

Cited by 35 publications

References 13 publications

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Acceptance of AI-Based Meeting Tools: Psychological Safety as a Foundation for Smart Collaboration

See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People

Contact Info

Product

Resources

About