Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-Latent Space

Papastratis, Ilias; Dimitropoulos, Kosmas; Konstantinidis, Dimitrios; Daras, Petros

doi:10.1109/access.2020.2993650

Cited by 61 publications

(31 citation statements)

References 33 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The inception network and self-attention networks were optimized jointly with clip-level feature learning and sequence learning. On the other hand, in [ 7 ], a cross-modal approach was proposed for RGB-based CSLR. The extracted video and text representations were aligned into a joint latent space while a jointly trained decoder was employed.…”

Section: Related Workmentioning

confidence: 99%

“…This is mainly because sign languages feature thousands of signs, sometimes differing only by subtle changes in hand motion, shape, or position and involving significant finger overlaps and occlusions [ 2 ]. SLR tasks are divided into Isolated Sign Language Recognition (ISLR) [ 3 , 4 , 5 ] and Continuous Sign Language Recognition (CSLR) [ 6 , 7 , 8 ]. The CSLR task focuses on recognizing sequences of glosses from videos without predefined annotation boundaries, and it is more challenging compared to ISLR [ 9 ], in which the temporal boundaries of glosses in the videos are predefined.…”

Section: Introductionmentioning

confidence: 99%

“…They adopted various techniques for extracting handcrafted features [ 10 ], e.g., hand and joint trajectories [ 11 ] or histograms of oriented gradients [ 12 ], while employing traditional sequence-modeling mechanisms, such as Hidden Markov Models (HMMs) [ 13 , 14 ] or Conditional Random Fields (CRFs) [ 15 ], for the modeling of the time-evolving information of a sign. On the other hand, recent SLR methods have achieved outstanding performance using deep-learning networks [ 7 , 16 , 17 ]—in particular, convolutional neural networks (CNNs) and recurrent neural networks (RNNs)—to efficiently extract visual and temporal information from video sequences. Two-dimensional CNNs have already proven their ability to model complex or even heterogeneous data [ 18 ], while 3D-CNN or CNN–RNN network architectures have been widely used for spatio-temporal modeling in a variety of applications [ 19 , 20 , 21 ].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Papastratis

Dimitropoulos

Daras

2021

Sensors

Self Cite

View full text Add to dashboard Cite

Continuous sign language recognition is a weakly supervised task dealing with the identification of continuous sign gestures from video sequences, without any prior knowledge about the temporal boundaries between consecutive signs. Most of the existing methods focus mainly on the extraction of spatio-temporal visual features without exploiting text or contextual information to further improve the recognition accuracy. Moreover, the ability of deep generative models to effectively model data distribution has not been investigated yet in the field of sign language recognition. To this end, a novel approach for context-aware continuous sign language recognition using a generative adversarial network architecture, named as Sign Language Recognition Generative Adversarial Network (SLRGAN), is introduced. The proposed network architecture consists of a generator that recognizes sign language glosses by extracting spatial and temporal features from video sequences, as well as a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and gloss levels. The paper also investigates the importance of contextual information on sign language conversations for both Deaf-to-Deaf and Deaf-to-hearing communication. Contextual information, in the form of hidden states extracted from the previous sentence, is fed into the bidirectional long short-term memory module of the generator to improve the recognition accuracy of the network. At the final stage, sign language translation is performed by a transformer network, which converts sign language glosses to natural language text. Our proposed method achieved word error rates of 23.4%, 2.1%, and 2.26% on the RWTH-Phoenix-Weather-2014 and the Chinese Sign Language (CSL) and Greek Sign Language (GSL) Signer Independent (SI) datasets, respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Papastratis

Dimitropoulos

Daras

2021

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Accurate hand pose estimation can enhance user experience in VR systems by enabling the performance of plausible and realistic virtual hand movements, as well as contribute towards a better understanding of human actions in smart HCI systems [5][6][7][8], thus enabling a more intelligent interaction between users and smart systems. Apart from the aforementioned applications, hand pose estimation is crucial in a number of other tasks, such as gesture recognition [9,10], action recognition [11,12], support systems for motor impairments patients [13], sign language recognition [14][15][16][17][18], and representation [19]. In sign language recognition especially, accurate hand pose estimation is beneficial for promoting social inclusion and enhancing accessibility in the Deaf community.…”

Section: Introductionmentioning

confidence: 99%

A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods

et al. 2020

Self Cite

View full text Add to dashboard Cite

The field of 3D hand pose estimation has been gaining a lot of attention recently, due to its significance in several applications that require human-computer interaction (HCI). The utilization of technological advances, such as cost-efficient depth cameras coupled with the explosive progress of Deep Neural Networks (DNNs), has led to a significant boost in the development of robust markerless 3D hand pose estimation methods. Nonetheless, finger occlusions and rapid motions still pose significant challenges to the accuracy of such methods. In this survey, we provide a comprehensive study of the most representative deep learning-based methods in literature and propose a new taxonomy heavily based on the input data modality, being RGB, depth, or multimodal information. Finally, we demonstrate results on the most popular RGB and depth-based datasets and discuss potential research directions in this rapidly growing field.

show abstract

“…As a result, recognition accuracy could be improved to a WER of

. Lastly, Papastratis et al [ 25 ] showed that that the accuracy of sentence predictions could be further enhanced by a cross-modal learning approach that leverages text information.…”

Section: Introductionmentioning

confidence: 99%

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Brock

Farag

Nakadai

2020

Sensors

View full text Add to dashboard Cite

The quality of recognition systems for continuous utterances in signed languages could be largely advanced within the last years. However, research efforts often do not address specific linguistic features of signed languages, as e.g., non-manual expressions. In this work, we evaluate the potential of a single video camera-based recognition system with respect to the latter. For this, we introduce a two-stage pipeline based on two-dimensional body joint positions extracted from RGB camera data. The system first separates the data flow of a signed expression into meaningful word segments on the base of a frame-wise binary Random Forest. Next, every segment is transformed into image-like shape and classified with a Convolutional Neural Network. The proposed system is then evaluated on a data set of continuous sentence expressions in Japanese Sign Language with a variation of non-manual expressions. Exploring multiple variations of data representations and network parameters, we are able to distinguish word segments of specific non-manual intonations with 86% accuracy from the underlying body joint movement data. Full sentence predictions achieve a total Word Error Rate of 15.75%. This marks an improvement of 13.22% as compared to ground truth predictions obtained from labeling insensitive towards non-manual content. Consequently, our analysis constitutes an important contribution for a better understanding of mixed manual and non-manual content in signed communication.

show abstract

Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-Latent Space

Cited by 61 publications

References 33 publications

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Contact Info

Product

Resources

About