Spoken Spanish generation from sign language

Abstract

a b s t r a c tThis paper describes the development of a Spoken Spanish generator from sign-writing. The sign language considered was the Spanish sign language (LSE: Lengua de Signos Española). This system consists of an advanced visual interface (where a deaf person can specify a sequence of signs in signwriting), a language translator (for generating the sequence of words in Spanish), and finally, a text to speech converter. The visual interface allows a sign sequence to be defined using several sign-writing alternatives. The paper details the process for designing the visual interface proposing solutions for HCI-specific challenges when working with the Deaf (i.e. important difficulties in writing Spanish or limited sign coverage for describing abstract or conceptual ideas). Three strategies were developed and combined for language translation to implement the final version of the language translator module. The summative evaluation, carried out with Deaf from Madrid and Toledo, includes objective measurements from the system and subjective information from questionnaires. The paper also describes the first Spanish-LSE parallel corpus for language processing research focused on specific domains. This corpus includes more than 4000 Spanish sentences translated into LSE. These sentences focused on two restricted domains: the renewal of the identity document and driver's license. This corpus also contains all sign descriptions in several sign-writing specifications generated with a new version of the eSign Editor. This new version includes a grapheme to phoneme system for Spanish and a SEA-HamNoSys converter.

Design, development and field evaluation of a Spanish into sign language translation system

“…The sign database has been generated using a new version [34] of the eSIGN Editor [14]. The eSIGN Editor was developed in the VISICAST and eSIGN European Projects (Essential Sign Language Information on Government Networks).…”

“…The eSIGN Editor was developed in the VISICAST and eSIGN European Projects (Essential Sign Language Information on Government Networks). In a previous work [34], this editor has been adapted to LSE. The new version incorporates the same functionality for defining manual movements (using HamNoSys and SEA) and non-manual aspects such as movements of lips, head, etc.…”

“…This sequence is necessary to make the avatar move the lips according to this pronunciation (see [34] for more details). Figure 11 shows the process for specifying a sign: defining the manual part (using HamNoSys or SEA), adding non-manual characteristics (lips, head and facial movements) and generating a script in the Sign Gesture Markup Language (SiGML) (XML file).…”

“…In order to allow Deaf people to ask questions to the government officer, a spoken Spanish generation system has been used [34]. With this system, the deaf person can specify a sign sequence (gloss sequence) with a set of visual tools.…”

“…As will be presented in next section, there are similar systems for other languages, but the system described here is the first one for translating Spanish into LSE evaluated involving real interactions between Deaf and hearing people without an interpreter: government employees that provide a service (renewing a DL) and Deaf people that want to access this service. The proposed system translates the government employee's explanations into LSE for Deaf people and they can ask questions to the government employee using a spoken Spanish generation system from gloss sequences [34]. This paper is organised as follows.…”