Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech

Abstract

This article is devoted to solving the problem of converting sign language into a consistent text with intonation markup for subsequent voice synthesis of sign phrases by speech with intonation. The paper proposes an improved method of continuous recognition of sign language, the results of which are transmitted to a natural language processor based on analyzers of morphology, syntax, and semantics of the Kazakh language, including morphological inflection and the construction of an intonation model of simple sentences. This approach has significant practical and social significance, as it can lead to the development of technologies that will help people with disabilities to communicate and improve their quality of life. As a result of the cross-validation of the model, we obtained an average test accuracy of 0.97 and an average val_accuracy of 0.90 for model evaluation. We also identified 20 sentence structures of the Kazakh language with their intonational model.

Efhamni: A Deep Learning-Based Saudi Sign Language Recognition Application

“…Additionally, sign language recognition systems can be classified into either continuous or isolated. In isolated sign language translation systems, only one sign gesture is displayed and translated at a time, while continuous sign language translation systems offer a complete clause translation [5,6]. While isolated sign language translation systems do not support complete sentences, they can aid in improving technologies for recognizing and translating individual signs, which is necessary for building more complex systems in the future that can handle phrases and sentences.…”

“…As a result, there are currently no fully automated models and methods for recognizing several static and, most importantly, dynamic gestures. To develop such automated models, deep semantic analysis is required, which is challenging due to the limited number and the small size of sign language datasets [5]. Few research studies, however, have targeted the ArSL due to its complexity compared to the non-Arabic SL and the data unavailability (e.g., [9]), and even fewer have targeted the Saudi Sign Language (SSL) (e.g., [3,[10][11][12][13][14][15]).…”