Automatic fingersign-to-speech translation system

Hrúz, Marek; Campr, Pavel; Dikici, Erinç; Kındıroğlu, Ahmet Alp; Krňoul, Zdeněk; Ronzhin, Andrey; Sak, Haşim; Schorno, Daniel; Yalçın, Hülya; Akarun, Lale; Aran, Oya; Karpov, Alexey; Saraçlar, Murat; Železný, Miloš

doi:10.1007/s12193-011-0059-3

Cited by 8 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although signs may be defined differently, how signs are performed can be categorized into a few signing schemes. American, French, and Russian Sign Languages [1][2][3][4] employ a one-hand scheme; Australian and British Sign Languages [5,6] employed a two-hand scheme. To cope with 42 alphabets and 20 vowels, TFS employs a one-hand scheme with an extension using movement and multi-posture signings for the alphabets and a two-hand scheme for the vowels and intonation marks [7].…”

Section: Introductionmentioning

confidence: 99%

Automatic Thai Finger Spelling Transcription

Nakjai

Katanyukul

2021

Walailak J Sci & Tech

View full text Add to dashboard Cite

This article explores a transcription of a video recording Thai Finger Spelling (TFS)—a specific signing mode used in Thai sign language—to a corresponding Thai word. TFS copes with 42 Thai alphabets and 20 vowels using multiple and complex schemes. This leads to many technical challenges uncommon in spelling schemes of other sign languages. Our proposed system, Automatic Thai Finger Spelling Transcription (ATFS), processes a signing video in 3 stages: ALS marking video frames to easily remove any non-signing frame as well as conveniently group frames associating to the same alphabet, SR classifying a signing image frame to a sign label (or its equivalence), and SSR transcribing a series of signs into alphabets. ALS utilizes the TFS practice of signing different alphabets at different locations. SR and SSC employ well-adopted spatial and sequential models. Our ATFS has been found to achieve Alphabet Error Rate (AER) 0.256 (c.f. 0.63 of the baseline method). In addition to ATFS, our findings have disclosed a benefit of coupling image classification and sequence modeling stages by using a feature or penultimate vector for label representation rather than a definitive label or one-hot coding. Our results also assert the necessity of a smoothening mechanism in ALS and reveal a benefit of our proposed WFS, which could lead to over 15.88 % improvement. For TFS transcription, our work emphasizes the utilization of signing location in the identification of different alphabets. This is contrary to a common belief of exploiting signing time duration, which are shown to be ineffective by our data. HIGHLIGHTS Prototype of Thai finger spelling transcription (transcribing a signing video to alphabets) Utilization of signing location as cue for identification of different alphabets Disclosure of a benefit of coupling image classification and sequence modeling in signing transcription Examination of various frame smoothing techniques and their contributions to the overall transcription performance GRAPHICAL ABSTRACT

show abstract

Section: Introductionmentioning

confidence: 99%

Automatic Thai Finger Spelling Transcription

Nakjai

Katanyukul

2021

Walailak J Sci & Tech

View full text Add to dashboard Cite

show abstract

“…However, universal assistive technologies are rarely developed. One example of such technology is a computer system that supports natural communication between blind and deaf persons [5,6]. Some other examples of multimodal human-computer interfaces for universal access framework are presented in [7][8][9][10][11].…”

Section: Introductionmentioning

confidence: 99%

A Universal Assistive Technology with Multimodal Input and Multimedia Output Interfaces

Karpov

Ronzhin

2014

Universal Access in Human-Computer Interaction. Design and Development Methods for Universal Access

Self Cite

View full text Add to dashboard Cite

In this paper, we present a universal assistive technology with multimodal input and multimedia output interfaces. The conceptual model and the software-hardware architecture with levels and components of the universal assistive technology are described. The architecture includes five main interconnected levels: computer hardware, system software, application software of digital signal processing, application software of human-computer interfaces, software of assistive information technologies. The universal assistive technology proposes several multimodal systems and interfaces to the people with disabilities: audio-visual Russian speech recognition system (AVSR), "Talking head" synthesis system (text-to-audiovisual speech), "Signing avatar" synthesis system (sign language visual synthesis), ICANDO multimodal system (handsfree PC control system), and the control system of an assistive smart space.

show abstract