Temporal Lift Pooling for Continuous Sign Language Recognition

Hu, Lianyu; Gao, Liqing; Liu, Zekang; Feng, Wei

doi:10.1007/978-3-031-19833-5_30

Cited by 19 publications

(10 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng [22] The paper discusses the significance of sign language as a communication tool for disabled individuals and the challenges hearing people face in learning it. It introduces vision-based continuous sign language recognition (CSLR) as a means to bridge the communication gap between the two groups by automatically translating sign videos into sentences.…”

Section: Literature Surveymentioning

confidence: 99%

Reverse Sign Language Recognition

Gaikwad

2024

IJSREM

View full text Add to dashboard Cite

Sign language is an important mode of communication for deaf, mute, and disabled people. Sign language is a language in which people communicate with the help of gestures in situations where they cannot speak or hear. Gestures are an effective method of human interaction and are often used by deaf people to communicate. This article describes various methods and techniques for recognizing text signatures in images. This review compares different methods and algorithms with the help of a pie chart. Key Words: communication, sign recognition, hand gestures, sign language

show abstract

Section: Literature Surveymentioning

confidence: 99%

Reverse Sign Language Recognition

Gaikwad

2024

IJSREM

View full text Add to dashboard Cite

show abstract

“…While the prior SL literature focuses more on techniques such as Hidden Markov Models (HMMs) for sequence modeling after extracting handcrafted features, recent studies follow the idea of employing 2D-3D CNN and RNN-based architectures in which frames or skeleton joint information are directly used (Aran, 2008 ; Camgöz et al, 2016a ; Koller et al, 2016 , 2019 ; Zhang et al, 2016 ; Mittal et al, 2019 ; Abdullahi and Chamnongthai, 2022 ; Samaan et al, 2022 ). More recently, Transformer based architectures have become popular on SLR and Sign Language Translation (SLT) tasks due to their success in domains such as Natural Language Processing (NLP) and Speech Processing (SP) (Vaswani et al, 2017 ; Camgoz et al, 2020b ; Rastgoo et al, 2020 ; Boháček and Hrúz, 2022 ; Cao et al, 2022 ; Chen et al, 2022 ; Hrúz et al, 2022 ; Hu et al, 2022 ; Xie et al, 2023 ).…”

Section: Related Workmentioning

confidence: 99%

Multi-cue temporal modeling for skeleton-based sign language recognition

2023

View full text Add to dashboard Cite

Sign languages are visual languages used as the primary communication medium for the Deaf community. The signs comprise manual and non-manual articulators such as hand shapes, upper body movement, and facial expressions. Sign Language Recognition (SLR) aims to learn spatial and temporal representations from the videos of the signs. Most SLR studies focus on manual features often extracted from the shape of the dominant hand or the entire frame. However, facial expressions combined with hand and body gestures may also play a significant role in discriminating the context represented in the sign videos. In this study, we propose an isolated SLR framework based on Spatial-Temporal Graph Convolutional Networks (ST-GCNs) and Multi-Cue Long Short-Term Memorys (MC-LSTMs) to exploit multi-articulatory (e.g., body, hands, and face) information for recognizing sign glosses. We train an ST-GCN model for learning representations from the upper body and hands. Meanwhile, spatial embeddings of hand shape and facial expression cues are extracted from Convolutional Neural Networks (CNNs) pre-trained on large-scale hand and facial expression datasets. Thus, the proposed framework coupling ST-GCNs with MC-LSTMs for multi-articulatory temporal modeling can provide insights into the contribution of each visual Sign Language (SL) cue to recognition performance. To evaluate the proposed framework, we conducted extensive analyzes on two Turkish SL benchmark datasets with different linguistic properties, BosphorusSign22k and AUTSL. While we obtained comparable recognition performance with the skeleton-based state-of-the-art, we observe that incorporating multiple visual SL cues improves the recognition performance, especially in certain sign classes where multi-cue information is vital. The code is available at: https://github.com/ogulcanozdemir/multicue-slr.

show abstract

“…Some recent studies [5,17,34] try to directly enhance the feature extractor by adding alignment losses [17,34] or adopt pseudo labels [5] in a lightweight way, alleviating the heavy computational burden. More recent works enhance CSLR by squeezing more representative temporal features [22] or dynamically emphasizing informative spatial regions [23].…”

Section: Continuous Sign Language Recognitionmentioning

confidence: 99%

Continuous Sign Language Recognition with Correlation Network

Hu¹,

Gao²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Human body trajectories are a salient cue to identify actions in the video. Such body trajectories are mainly conveyed by hands and face across consecutive frames in sign language. However, current methods in continuous sign language recognition (CSLR) usually process frames independently, thus failing to capture cross-frame trajectories to effectively identify a sign. To handle this limitation, we propose correlation network (CorrNet) to explicitly capture and leverage body trajectories across frames to identify signs. In specific, a correlation module is first proposed to dynamically compute correlation maps between the current frame and adjacent frames to identify trajectories of all spatial patches. An identification module is then presented to dynamically emphasize the body trajectories within these correlation maps. As a result, the generated features are able to gain an overview of local temporal movements to identify a sign. Thanks to its special attention on body trajectories, CorrNet achieves new state-of-the-art accuracy on four large-scale datasets, i.e., PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the effectiveness of CorrNet. Visualizations demonstrate the effects of CorrNet on emphasizing human body trajectories across adjacent frames.

show abstract

Temporal Lift Pooling for Continuous Sign Language Recognition

Cited by 19 publications

References 28 publications

Reverse Sign Language Recognition

Reverse Sign Language Recognition

Multi-cue temporal modeling for skeleton-based sign language recognition

Continuous Sign Language Recognition with Correlation Network

Contact Info

Product

Resources

About