Considering multi-modal speech visualization for deaf and hard of hearing people

Abstract

Supporting deaf and hard of hearing (D/HH) people to understand natural conversation is one of the important activities of social welfare. However, currently the communication support for D/HH people is not enough in Japan. Although existing communication methods, such as sign language and lipreading, are effective in one-to-one conversation, they have several disadvantages in one-to-many conversation such as meetings or conventions. In order to support D/HH people in understanding conversation, this paper proposes a multi-modal visualization application which provides many aspects of information about speech contents. Concrete examples of visualization modes include displaying subtitles by voice recognition and showing speaker's mouth to assist lip-reading.

Discussion-facilitator: towards enabling students with hearing disabilities to participate in classroom discussions

“…Often, the sign language for a given spoken language is not standardised, which makes it hard for deaf students to take advantage from such systems during class (Toba et al, 2015). Moreover, the translation, in general, is not fluent, as there are several sources of errors as well as complex components (such as facial expressions) that also need to be translated.…”

“…Thus, fluent speakers of any language can lip read, to some degree (Nath and Beauchamp, 2012). Some deaf or hard of hearing people become highly proficient at lip reading, which helps them understand speech from the lip movements of the speaker (Toba et al, 2015).…”

“…Several studies have shown that if people who are hearing impaired are able to closely view lip and tongue movements (as well as the facial expressions of a speaker) they are better able to recognise, interpret and understand what is being spoken (Toba et al, 2015).…”

A new technology on translating Indonesian spoken language into Indonesian sign language system

“…This condition might be either temporary or permanent. For those living around people with hearing disabilities, a special communication form is highly needed in order to properly deliver the message in a conversation [1][2][3]. Indonesian sign language system or sistem isyarat Bahasa Indonesia (SIBI) is a sign language system specifically developed for people with hearing disabilities.…”