FluentSigners-50: A signer independent benchmark dataset for sign language processing

Mukushev, Medet; Ubingazhibov, Aidyn; Kydyrbekova, Aigerim; Imashev, Alfarabi; Kimmelman, Vadim; Sandygulova, Anara

doi:10.1371/journal.pone.0273649

Cited by 4 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Many approaches to model aspects of sign language have been described in the literature. Hand-shape classification methods attempt to recognise specific configurations of hands [4,16,[45][46][47][48], which can be considered the fundamental sign articulator-this has particular importance for recognising finger spelling [49]. Others focus on modelling nonmanual aspects, which are essential context carriers, not least for discriminating between signs with similar appearance.…”

Section: Modellingmentioning

confidence: 99%

Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data

Woods

Rana

2023

Mathematics

View full text Add to dashboard Cite

We present a study on modelling American Sign Language (ASL) with encoder-only transformers and human pose estimation keypoint data. Using an enhanced version of the publicly available Word-level ASL (WLASL) dataset, and a novel normalisation technique based on signer body size, we show the impact model architecture has on accurately classifying sets of 10, 50, 100, and 300 isolated, dynamic signs using two-dimensional keypoint coordinates only. We demonstrate the importance of running and reporting results from repeated experiments to describe and evaluate model performance. We include descriptions of the algorithms used to normalise the data and generate the train, validation, and test data splits. We report top-1, top-5, and top-10 accuracy results, evaluated with two separate model checkpoint metrics based on validation accuracy and loss. We find models with fewer than 100k learnable parameters can achieve high accuracy on reduced vocabulary datasets, paving the way for lightweight consumer hardware to perform tasks that are traditionally resource-intensive, requiring expensive, high-end equipment. We achieve top-1, top-5, and top-10 accuracies of 97%, 100%, and 100%, respectively, on a vocabulary size of 10 signs; 87%, 97%, and 98% on 50 signs; 83%, 96%, and 97% on 100 signs; and 71%, 90%, and 94% on 300 signs, thereby setting a new benchmark for this task.

show abstract

Section: Modellingmentioning

confidence: 99%

Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data

Woods

Rana

2023

Mathematics

View full text Add to dashboard Cite

show abstract

“…Современные работы по распознаванию жестового языка сталкиваются с такими проблемами, как, например, определение границ элемента языка, а именно -является ли движение продолжением одного жеста или уже началом следующего [123]. Эта же проблема является препятствием для автоматической разметки наборов данных, как в [124].…”

Section: рис 3 основные методы автоматического распознавания устной р...unclassified

Modern automatic recognition technologies for visual communication tools

Yachnaya¹,

Lutsiv²,

Malashin³

2023

Computer Optics

View full text Add to dashboard Cite

Communication refers to a wide range of different behaviors and activities aimed at handing over information. The communication process includes verbal, paraverbal and non-verbal components, conveying the informational part of a message and its emotional part respectively. A complex analysis of all communication components makes it possible to evaluate not only the content, but also the situational context of what is being said, as well as to identify additional factors inherent in the mental and somatic state of the speaker. There are several methods of conveying a verbal message, among which are oral and gestural speech (such as the sign language and fingerspelling). Various forms of communication can be contained in multiple data transmission channels, such as audio or video channels. The review is concerned with video data analysis systems, as the audio channel is incapable of non-verbal components transmission that contribute supplemental details. The article analyzes databases of static and dynamic images and systems, developed to recognize the verbal component conveyed by oral and gestural speech, as well as systems that evaluate paraverbal and non-verbal components of communication. Challenges of designing such databases and systems are specified. Prospective directions in complex analysis of all communication components and its combinations for the most complete evaluation of messages are defined.

show abstract