A Deep Learning Approach for Analyzing Video and Skeletal Features in Sign Language Recognition

Konstantinidis, Dimitrios; Dimitropoulos, Kosmas; Daras, Petros

doi:10.1109/ist.2018.8577085

Cited by 43 publications

(32 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These details make the problem complicated, as it is not essentially a full repetitive set of actions that mean the same thing, but a constructed discourse [25] with verbs and actions, as well as facial expressions, including happiness, deception, disappointment, and other impressions that can be driven in synchronization with the hand movements. Authors of [26], have recapitulated, in a very organized way, all the methods categorizing sign language over a full pipeline methodology recognition process, from the sensor acquisition to processing, and then to recognition methods.…”

Section: Methods Used In Arslmentioning

confidence: 99%

Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data

et al. 2021

View full text Add to dashboard Cite

This paper presents a novel Arabic Sign Language (ArSL) recognition system, using selected 2D hands and body key points from successive video frames. The system recognizes the recorded video signs, for both signer dependent and signer independent modes, using the concatenation of a 3D CNN skeleton network and a 2D point convolution network. To accomplish this, we built a new ArSL video-based sign database. We will present the detailed methodology of recording the new dataset, which comprises 80 static and dynamic signs that were repeated five times by 40 signers. The signs include Arabic alphabet, numbers, and some daily use signs. To facilitate building an online sign recognition system, we introduce the inverse efficiency score to find a sufficient optimal number of successive frames for the recognition decision, in order to cope with a near real-time automatic ArSL system, where tradeoff between accuracy and speed is crucial to avoid delayed sign classification. For the dependent mode, best results were obtained for dynamic signs with an accuracy of 98.39%, and 88.89% for the static signs, and for the independent mode, we obtained for the dynamic signs an accuracy of 96.69%, and 86.34% for the static signs. When both the static and dynamic signs were mixed and the system trained with all the signs, accuracies of 89.62% and 88.09% were obtained in the signer dependent and signer independent modes respectively.

show abstract

Section: Methods Used In Arslmentioning

confidence: 99%

Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Input observations are skeleton joint information, depth and RGB images, using a Gauss-Bernoulli deep belief network suitable for input forms to process bone dynamics, and applying convolutional neural networks to adjust and fuse batch depth and RGB images. Konstantinidis et al [16] propose a sign language recognition RNN networks based on RGB, skeleton data, and facial expression features. The data fusion schemes are analyzed.…”

Section: A Rgb-dmentioning

confidence: 99%

Multimodal Spatiotemporal Networks for Sign Language Recognition

Zhang

Meng

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Different from other human behaviors, sign language has the characteristics of limited local motion of upper limb and meticulous hand action. Some sign language gestures are ambiguous in RGB video due to the influence of lighting and background color, which affects the recognition accuracy. We propose a multimodal deep learning architecture for sign language recognition which effectively combines RGB-D input and two-stream spatiotemporal networks. Depth videos, as an effective compensation of RGB input, can supply additional distance information about the signer's hands. A novel sampling method called ARSS (Aligned Random Sampling in Segments) is put forward to select and align optimal RGB-D video frames, which improves the capacity utilization of multimodal data and reduces the redundancy. We get the hand ROI by joints information of RGB data for local focus in spatial stream. D-shift Net is proposed as depth motion feature extraction in temporal stream, which fully utilizes three dimensional motion information of the sign language. Both streams are fused by convolutional fusion layer to get complementary features. Our approach explored the multimodal information and enhanced the recognition precision. It obtains the state-the-of-art performance on the datasets of CSL (96.7%) and IsoGD (63.78%). INDEX TERMS Sign language recognition, two-stream network, motion features, multimodal data.

show abstract

“…This is mainly because sign languages feature thousands of signs, sometimes differing only by subtle changes in hand motion, shape, or position and involving significant finger overlaps and occlusions [ 2 ]. SLR tasks are divided into Isolated Sign Language Recognition (ISLR) [ 3 , 4 , 5 ] and Continuous Sign Language Recognition (CSLR) [ 6 , 7 , 8 ]. The CSLR task focuses on recognizing sequences of glosses from videos without predefined annotation boundaries, and it is more challenging compared to ISLR [ 9 ], in which the temporal boundaries of glosses in the videos are predefined.…”

Section: Introductionmentioning

confidence: 99%

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Papastratis

Dimitropoulos

Daras

2021

Sensors

Self Cite

View full text Add to dashboard Cite

Continuous sign language recognition is a weakly supervised task dealing with the identification of continuous sign gestures from video sequences, without any prior knowledge about the temporal boundaries between consecutive signs. Most of the existing methods focus mainly on the extraction of spatio-temporal visual features without exploiting text or contextual information to further improve the recognition accuracy. Moreover, the ability of deep generative models to effectively model data distribution has not been investigated yet in the field of sign language recognition. To this end, a novel approach for context-aware continuous sign language recognition using a generative adversarial network architecture, named as Sign Language Recognition Generative Adversarial Network (SLRGAN), is introduced. The proposed network architecture consists of a generator that recognizes sign language glosses by extracting spatial and temporal features from video sequences, as well as a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and gloss levels. The paper also investigates the importance of contextual information on sign language conversations for both Deaf-to-Deaf and Deaf-to-hearing communication. Contextual information, in the form of hidden states extracted from the previous sentence, is fed into the bidirectional long short-term memory module of the generator to improve the recognition accuracy of the network. At the final stage, sign language translation is performed by a transformer network, which converts sign language glosses to natural language text. Our proposed method achieved word error rates of 23.4%, 2.1%, and 2.26% on the RWTH-Phoenix-Weather-2014 and the Chinese Sign Language (CSL) and Greek Sign Language (GSL) Signer Independent (SI) datasets, respectively.

show abstract

A Deep Learning Approach for Analyzing Video and Skeletal Features in Sign Language Recognition

Cited by 43 publications

References 23 publications

Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data

Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data

Multimodal Spatiotemporal Networks for Sign Language Recognition

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Contact Info

Product

Resources

About