A method of multimodal machine sign language translation for natural human-computer interaction

Axyonov, Alexandr; Kagirov, Ildar; Ryumin, Dmitry

doi:10.17586/2226-1494-2022-22-3-585-593

Cited by 2 publications

(4 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ROIs are shown in Figure 7 c. The face region is fed to pre-trained models ( accessed on 6 February 2023) from the Deepface open source software platform [ 135 , 136 ] for machine classification of the signer’s gender and age. Previously, we used these models in a similar problem of gesture recognition [ 137 ]. In Ref.…”

Section: Methodsmentioning

confidence: 99%

“…This strategy is called transfer learning and has proven effective in other CV problems [ 120 ]. Finally, we extract NN hand representations using the E2Ev2 [ 137 ] model.…”

Section: Methodsmentioning

confidence: 99%

“…Experiments to optimize the gesture recognition model are focused on the selection of a set of spatio-temporal features. As basic features, we use the [ 137 ] features that have proven themselves in our previous study: 2D distances from face to hands (two features per frame); Areas of face and hands intersection (two features per frame); Zones of hands location (two features per frame); Age estimate (one feature per frame); Gender estimate (one feature per frame). …”

Section: Evaluation Experimentsmentioning

confidence: 99%

“…To these features, we add hand configuration features extracted using the E2Ev2 [ 137 ] neural model. Each hand has its own NN features with the size of 512.…”

Section: Evaluation Experimentsmentioning

confidence: 99%

See 3 more Smart Citations

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Ryumin

Ivanko

Ryumina

2023

Sensors

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human–computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora—LRW and AUTSL—and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

show abstract

Section: Methodsmentioning

confidence: 99%

“…This strategy is called transfer learning and has proven effective in other CV problems [ 120 ]. Finally, we extract NN hand representations using the E2Ev2 [ 137 ] model.…”

Section: Methodsmentioning

confidence: 99%

Section: Evaluation Experimentsmentioning

confidence: 99%

“…To these features, we add hand configuration features extracted using the E2Ev2 [ 137 ] neural model. Each hand has its own NN features with the size of 512.…”

Section: Evaluation Experimentsmentioning

confidence: 99%

See 2 more Smart Citations

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Ryumin

Ivanko

Ryumina

2023

Sensors

View full text Add to dashboard Cite

show abstract

Machine translation from signed to spoken languages: state of the art and challenges

Coster

Shterionov

Herreweghe

et al. 2023

Univ Access Inf Soc

View full text Add to dashboard Cite

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity—the majority of scientific papers on sign language (SL) translation have been published in the past five years—research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

show abstract

A method of multimodal machine sign language translation for natural human-computer interaction

Cited by 2 publications

References 5 publications

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Machine translation from signed to spoken languages: state of the art and challenges

Contact Info

Product

Resources

About