Global-Local Enhancement Network for NMF-Aware Sign Language Recognition

Hu, Hezhen; Zhou, Wengang; Pu, Junfu; Li, Houqiang

doi:10.1145/3436754

Cited by 40 publications

(32 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observe that our approach can preserve fine-grained identity attributes of the source image, such as face and hair. In the future, we will continue exploring the potential of applying this method to other relevant research fields, such as data augmentation for sign language recognition [11], [66]- [70] and clothing / makeup try-on according to keypoints [71]- [73].…”

Section: Discussionmentioning

confidence: 99%

Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation

Suo¹,

Zheng²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Sign language is the window for people differentlyabled to express their feelings as well as emotions. However, it remains challenging for people to learn sign language in a short time. To address this real-world challenge, in this work, we study the motion transfer system, which can transfer the user photo to the sign language video of specific words. In particular, the appearance content of the output video comes from the provided user image, while the motion of the video is extracted from the specified tutorial video. We observe two primary limitations in adopting the state-of-the-art motion transfer methods to sign language generation: (1) Existing motion transfer works ignore the prior geometrical knowledge of the human body. (2) The previous image animation methods only take image pairs as input in the training stage, which could not fully exploit the temporal information within videos. In an attempt to address the above-mentioned limitations, we propose Structure-aware Temporal Consistency Network (STCNet) to jointly optimize the prior structure of human with the temporal consistency for sign language video generation. There are two main contributions in this paper. (1) We harness a fine-grained skeleton detector to provide prior knowledge of the body keypoints. In this way, we ensure the keypoint movement in a valid range and make the model become more explainable and robust. (2) We introduce two cycle-consistency losses, i.e., short-term cycle loss and long-term cycle loss, which are conducted to assure the continuity of the generated video. We optimize the two losses and keypoint detector network in an end-to-end manner. Empirical evaluation on the three widely-used datasets, i.e., LSA64, RWTH-PHOENIX-Weather 2014T, and WLASL-2000 verifies the effectiveness of the proposed method quantitatively and qualitatively. Besides, we also conduct extensive ablation studies to show that jointly learning the structure and temporal cues leads to high-fidelity video generation. We hope this work can pave the way for future studies on sign language generation, and social communication with fewer limitations.

show abstract

Section: Discussionmentioning

confidence: 99%

Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation

Suo¹,

Zheng²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…They used the MSASL dataset to transfer the ASL knowledge to recognize GSL (German Sign Language) on the SIGNUM dataset and achieved an accuracy of 0.75 for high target data. Hu et al [140] pointed out non-manual feature-aware GLEN (Global local enhancement network) based on the SLR model. They achieved a top 1 accuracy of 69.9% for NMFs-CSL datasets and 96.8% for isolated SLR 500 datasets.…”

Section: B Study Of Current State-of-the-art Models For Sign Language...mentioning

confidence: 99%

A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets

Madhiarasan¹,

Roy²

2022

Preprint

View full text Add to dashboard Cite

A machine can understand human activities, and the meaning of signs can help overcome the communication barriers between the inaudible and ordinary people. Sign Language Recognition (SLR) is a fascinating research area and a crucial task concerning computer vision and pattern recognition. Recently, SLR usage has increased in many applications, but the environment, background image resolution, modalities, and datasets affect the performance a lot. Many researchers have been striving to carry out generic real-time SLR models. This review paper facilitates a comprehensive overview of SLR and discusses the needs, challenges, and problems associated with SLR. We study related works about manual and non-manual, various modalities, and datasets. Research progress and existing stateof-the-art SLR models over the past decade have been reviewed. Finally, we find the research gap and limitations in this domain and suggest future directions. This review paper will be helpful for readers and researchers to get complete guidance about SLR and the progressive design of the state-of-the-art SLR model.

show abstract

“…cSLR aims at transcribing videobased sign language into gloss sequence. With the released of larger-scale cSLR datasets [28], numerous researches burst out implementing sign language recognition tasks in an end-to-end manner [1][2][3][4][5][6][7][8][9][10][11][12][13]. The gloss annotations are in same order with sign language, this monotonic relationship significantly ease the syntactic alignment with the cSLR methods.…”

Section: A Sign Language Recognitionmentioning

confidence: 99%

“…Sign Language Recognition (SLR) and Translation (SLT) aim at converting the video-based sign languages into sign gloss sequences and spoken language sentences, respectively. Most previous works in this field focus on continuous SLR with the gloss supervision [1][2][3][4][5][6][7][8][9][10][11][12][13], few attempts have been made for SLT [14][15][16][17]. The main difference is that gloss labels are in the same order with sign gestures, and thus the gloss annotations significantly ease the syntactic alignment under the SLR methods.…”

Section: Introductionmentioning

confidence: 99%

PiSLTRc: Position-informed Sign Language Transformer with Content-aware Convolution

Xie¹,

Zhao²,

Hu³

2021

Preprint

View full text Add to dashboard Cite

Since the superiority of Transformer in learning long-term dependency, the sign language Transformer model achieves remarkable progress in Sign Language Recognition (SLR) and Translation (SLT). However, there are several issues with the Transformer that prevent it from better sign language understanding. The first issue is that the self-attention mechanism learns sign video representation in a frame-wise manner, neglecting the temporal semantic structure of sign gestures. Secondly, the attention mechanism with absolute position encoding is direction and distance unaware, thus limiting its ability. To address these issues, we propose a new model architecture, namely PiSLTRc, with two distinctive characteristics: (i) contentaware and position-aware convolution layers. Specifically, we explicitly select relevant features using a novel content-aware neighborhood gathering method. Then we aggregate these features with position-informed temporal convolution layers, thus generating robust neighborhood-enhanced sign representation. (ii) injecting the relative position information to the attention mechanism in the encoder, decoder, and even encoder-decoder cross attention. Compared with the vanilla Transformer model, our model performs consistently better on three large-scale sign language benchmarks: PHOENIX-2014, PHOENIX-2014-T and CSL. Furthermore, extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on translation quality with +1.6 BLEU improvements.

show abstract

Global-Local Enhancement Network for NMF-Aware Sign Language Recognition

Cited by 40 publications

References 67 publications

Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation

Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation

A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets

PiSLTRc: Position-informed Sign Language Transformer with Content-aware Convolution

Contact Info

Product

Resources

About