Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning

Abstract

Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on semi-structured interviews and design probe activities we conducted with 15 DHH participants to understand their use of existing real-time captioning services and future design preferences for both in-person and remote small-group communication. We found that our participants' experiences of captioned small-group conversations are shaped by social, environmental, and technical considerations (e.g., interlocutors' pre-established relationships, the type of captioning displays available, and how far captions lag behind speech). When considering future captioning tools, participants were interested in greater feedback on non-speech elements of conversation (e.g., speaker identity, speech rate, volume) both for their personal use and to guide hearing interlocutors toward more accessible communication. We contribute a qualitative account of DHH people's real-time captioning experiences during small-group conversation and future design considerations to better support the groups being captioned, both in person and online.?

Sparkling Silence: Practices and Challenges of Livestreaming Among Deaf or Hard of Hearing Streamers

“…These include providing non-audible feedback [50], sign language support [2], and speechto-text translation [9]. Machine-generated or user-generated content, especially captions, has been shown to further improve accessibility for DHH internet users [25,30,40,53]. We know that both caption accuracy [25,30] and caption rate [58] have a strong impact on content comprehension.…”

The Connected Life: Using Access Technology at Home, at School and in the Community

“…Voice recognition software has been in development for many years; however, it was not until it was adopted by companies such as Apple for Siri that rapid improvements began to be seen. An interesting outcome of the COVID-19 pandemic has been the fact that many individuals without hearing loss have experienced difficulty with communication in online environments (e.g., Zoom meetings), and automated captioning technologies have begun to receive more attention by the general public [135][136][137][138][139]. Having companies such as Apple, Microsoft, Google and others focus their attention of research and development on voice recognition technology and its applications for captioning has resulted in exponential improvements in availability and accuracy, and this is expected to continue.…”

“…Attention is being paid to situations where captioning is needed, but effective technologies have not yet been developed, such as during student small group work or other activities where there are multiple speakers [136,137]. This also includes creating interfaces that make it easy for content creators to add captions to their own work, as knowledge and skills in this area are often lacking [138].…”

“…This also includes creating interfaces that make it easy for content creators to add captions to their own work, as knowledge and skills in this area are often lacking [138]. There is a new focus on including deaf individuals in the research and design stages of technology development, rather than simply presenting consumers with a finished product [115,137,139]. Virtual reality technology is now being investigated as a different method of providing captions in a real-time communication interaction with others via SpeechBubbles provided over the heads of speakers [140].…”