Sign Language Learning System with Image Sampling and Convolutional Neural Network

Ji, Yangho; Kim, Sunmok; Lee, Ki-Baek

doi:10.1109/irc.2017.40

Cited by 28 publications

(12 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, CNN has been widely used in human behavior recognition. As a representative deep learning network, CNN has a great improvement over the traditional neural network recognition effect [3][4][5][6][7][8][9]. Moreover, this method is an end-to-end recognition method, which does not need to be designed manually, and is of translation invariance and scale invariance.…”

Section: Introductionmentioning

confidence: 99%

[Retracted] Research on Webcast Supervision Based on Convolutional Neural Network and Wireless Communication

Sun

2021

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

Action recognition is the technology of understanding people’s behavior and classification from video or image sequences. This thesis uses the deep learning approach for action recognition to realize webcast supervision. This paper uses the convolutional neural network (CNN) and the Gaussian Mixture Model (GMM) to establish the webcast supervision system. At the same time, streaming-based wireless communication network technology is adopted to ensure video transmission speed and quality. Results show that the average detection speed of the system can reach 11.86 frame/s, and the average recognition accuracy is 92.16%, and the missed detection rate is lower than 5%. The design of this system can fully meet the requirements of webcast supervision.

show abstract

Section: Introductionmentioning

confidence: 99%

[Retracted] Research on Webcast Supervision Based on Convolutional Neural Network and Wireless Communication

Sun

2021

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

show abstract

“…Therefore, some research works use detection or tracking algorithms to extract the hand’s areas. Kim et al [ 30 , 31 ] used the target detection network to find the hands’ area and combined the original sign language data to feed the CNN network, which improved the accuracy and reduced the training time by half. Although traditional 2D CNN has strong feature extraction capabilities, it is not very suitable for the input of multi-frame image data.…”

Section: Related Workmentioning

confidence: 99%

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Lü

2021

Sensors

View full text Add to dashboard Cite

Sign language is the most important way of communication for hearing-impaired people. Research on sign language recognition can help normal people understand sign language. We reviewed the classic methods of sign language recognition, and the recognition accuracy is not high enough because of redundant information, human finger occlusion, motion blurring, the diversified signing styles of different people, and so on. To overcome these shortcomings, we propose a multi-scale and dual sign language recognition Network (SLR-Net) based on a graph convolutional network (GCN). The original input data was RGB videos. We first extracted the skeleton data from them and then used the skeleton data for sign language recognition. SLR-Net is mainly composed of three sub-modules: multi-scale attention network (MSA), multi-scale spatiotemporal attention network (MSSTA) and attention enhanced temporal convolution network (ATCN). MSA allows the GCN to learn the dependencies between long-distance vertices; MSSTA can directly learn the spatiotemporal features; ATCN allows the GCN network to better learn the long temporal dependencies. The three different attention mechanisms, multi-scale attention mechanism, spatiotemporal attention mechanism, and temporal attention mechanism, are proposed to further improve the robustness and accuracy. Besides, a keyframe extraction algorithm is proposed, which can greatly improve efficiency by sacrificing a little accuracy. Experimental results showed that our method can reach 98.08% accuracy rate in the CSL-500 dataset with a 500-word vocabulary. Even on the challenging dataset DEVISIGN-L with a 2000-word vocabulary, it also reached a 64.57% accuracy rate, outperforming other state-of-the-art sign language recognition methods.

show abstract

“…Rao et al 15 created a dataset containing over 200 signs in Indian sign language, but no data augmentation was used and the dataset was limited to signs performed with one hand. Ji et al 8 classified signs performing image sampling and using grayscale, achieving an accuracy of 86%, however just 6 signs were recognized, and no further exploration was done concerning data augmentation.…”

Section: Related Workmentioning

confidence: 99%

Efficient sign language recognition system and dataset creation method based on deep learning and image processing

Carneiro

Silva²,

Salvadeo

2021

Thirteenth International Conference on Digital Image Processing (ICDIP 2021)

View full text Add to dashboard Cite

New deep-learning architectures are created every year, achieving state-of-the-art results in image recognition and leading to the belief that, in a few years, complex tasks such as sign language translation will be considerably easier, serving as a communication tool for the hearing-impaired community. On the other hand, these algorithms still need a lot of data to be trained and the dataset creation process is expensive, time-consuming, and slow. Thereby, this work aims to investigate techniques of digital image processing and machine learning that can be used to create a sign language dataset effectively. We argue about data acquisition, such as the frames per second rate to capture or subsample the videos, the background type, preprocessing, and data augmentation, using convolutional neural networks and object detection to create an image classifier and comparing the results based on statistical tests. Different datasets were created to test the hypotheses, containing 14 words used daily and recorded by different smartphones in the RGB color system. We achieved an accuracy of 96.38% on the test set and 81.36% on the validation set containing more challenging conditions, showing that 30 FPS is the best frame rate subsample to train the classifier, geometric transformations work better than intensity transformations, and artificial background creation is not effective to model generalization. These trade-offs should be considered in future work as a cost-benefit guideline between computational cost and accuracy gain when creating a dataset and training a sign recognition model.

show abstract

Sign Language Learning System with Image Sampling and Convolutional Neural Network

Cited by 28 publications

References 10 publications

[Retracted] Research on Webcast Supervision Based on Convolutional Neural Network and Wireless Communication

[Retracted] Research on Webcast Supervision Based on Convolutional Neural Network and Wireless Communication

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Efficient sign language recognition system and dataset creation method based on deep learning and image processing

Contact Info

Product

Resources

About