Chinese Finger Sign Language Recognition Method with ResNet Transfer Learning

Chouvatut, Varin; Panyangam, Benjamas; Huang, Jiayu

doi:10.1109/kst57286.2023.10086825

Cited by 2 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Add the memory retained at the previous time in the network to the memory retained at the current time to obtain a new memory.The tanh layer g t generates a set of candidate values, which will be added to the storage unit if the input gate allows. According to (7), update the storage unit c t based on the output of the forget gate f t , input gate i t , and new candidate value g t . In formula (8), the output gate o t controls the state and memory information of the hidden state.…”

Section: Lstm Networkmentioning

confidence: 99%

“…In the sign language library, it can be divided into static sign language and dynamic sign language based on its state [6]. Static sign language mainly includes basic letters and some basic traffic signs; It is represented by the shape and direction of hands and fingers, which constitute the basic actions of sign language words [7]. The dynamic sign language consists mainly of hand movements and facial expressions, and therefore requires video streaming to record these combinations of movements.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Video-Based Sign Language Recognition via ResNet and LSTM Network

Huang,

Chouvatut

2024

Preprint

View full text Add to dashboard Cite

Sign language recognition technology can help people with hearing impairments to communicate with those who are hearing impaired. At present, with the rapid development of society, deep learning technology also provided certain technical support for sign language recognition work. In sign language recognition tasks, the use of traditional convolutional neural networks to extract spatio-temporal features from sign language videos suffers from insufficient feature extraction, resulting in low recognition rates. Nevertheless, video-based sign language datasets are very large and require a lot of computational resources for training and generalisation must be ensured, which poses a challenge for recognition. In this paper, we have presented a video-based sign language recognition method based on resnet and lstm. As the number of network layers increases, the ResNet network can effectively solve the granularity explosion problem and obtain better time series features. We use the Resnet convolutional network as the backbone model. At the initialisation stage, we obtain sign language features using ResNet; then, the learned feature space is used as the input of LSTM network to obtain long sequence features. The experimental results show that the accuracy of the above model is better than the mainstream model, and it can effectively extract the spatio-temporal features in sign language videos and improve the recognition rate of sign language actions.

show abstract

Section: Lstm Networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Video-Based Sign Language Recognition via ResNet and LSTM Network

Huang,

Chouvatut

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The tanh layer g t generates a set of candidate values, which will be added to the storage unit if the input gate allows. According to (7), update the storage unit c t based on the output of the forget gate f t , input gate i t , and new candidate value g t . In Formula (8), the output gate o t controls the state and memory information of the hidden state.…”

Section: Lstm Networkmentioning

confidence: 99%

“…Sign language can be divided into static sign language and dynamic sign language based on its state [ 6 ]. Static sign language mainly includes basic letters and some basic traffic signs; it is represented by the shape and direction of hands and fingers, which constitute the basic actions of sign language words [ 7 ]. Dynamic sign language consists mainly of hand movements and facial expressions and therefore requires video streaming to record these combinations of movements as shown Figure 1 .…”

Section: Introductionmentioning

confidence: 99%

Video-Based Sign Language Recognition via ResNet and LSTM Network

Huang,

Chouvatut

2024

J. Imaging

Self Cite

View full text Add to dashboard Cite

Sign language recognition technology can help people with hearing impairments to communicate with non-hearing-impaired people. At present, with the rapid development of society, deep learning also provides certain technical support for sign language recognition work. In sign language recognition tasks, traditional convolutional neural networks used to extract spatio-temporal features from sign language videos suffer from insufficient feature extraction, resulting in low recognition rates. Nevertheless, a large number of video-based sign language datasets require a significant amount of computing resources for training while ensuring the generalization of the network, which poses a challenge for recognition. In this paper, we present a video-based sign language recognition method based on Residual Network (ResNet) and Long Short-Term Memory (LSTM). As the number of network layers increases, the ResNet network can effectively solve the granularity explosion problem and obtain better time series features. We use the ResNet convolutional network as the backbone model. LSTM utilizes the concept of gates to control unit states and update the output feature values of sequences. ResNet extracts the sign language features. Then, the learned feature space is used as the input of the LSTM network to obtain long sequence features. It can effectively extract the spatio-temporal features in sign language videos and improve the recognition rate of sign language actions. An extensive experimental evaluation demonstrates the effectiveness and superior performance of the proposed method, with an accuracy of 85.26%, F1-score of 84.98%, and precision of 87.77% on Argentine Sign Language (LSA64).

show abstract

Chinese Finger Sign Language Recognition Method with ResNet Transfer Learning

Cited by 2 publications

References 11 publications

Video-Based Sign Language Recognition via ResNet and LSTM Network

Video-Based Sign Language Recognition via ResNet and LSTM Network

Video-Based Sign Language Recognition via ResNet and LSTM Network

Contact Info

Product

Resources

About