RGBD Video Based Human Hand Trajectory Tracking and Gesture Recognition System

Liu, Weihua; Fan, Yangyu; Li, Zuhe; Zhang, Zhong

doi:10.1155/2015/863732

Cited by 10 publications

(7 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental results have demonstrated that by fusing multiple channels in our proposed framework, the accuracy of recognizing ASL signs can be improved. This technology for identifying the appearance of specific ASL words has valuable applications for technologies that can benefit people who are DHH [9,42,43,48,52,65,68]. As an additional contribution, our "ASL-100-RGBD" dataset, will be released to enable other members of the research community to use this resource for training or evaluation of models for ASL recognition.…”

Section: Discussionmentioning

confidence: 99%

Recognizing American Sign Language Manual Signs from RGB-D Videos

Jing¹,

Huenerfauth²,

Tian³

2019

Preprint

View full text Add to dashboard Cite

In this paper, we propose a 3D Convolutional Neural Network (3DCNN) based multi-stream framework to recognize American Sign Language (ASL) manual signs (consisting of movements of the hands, as well as non-manual face movements in some cases) in real-time from RGB-D videos, by fusing multimodality features including hand gestures, facial expressions, and body poses from multi-channels (RGB, depth, motion, and skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3DCNN model. We collect a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera, each of 100 ASL manual signs, including RGB channel, depth maps, skeleton joints, face features, and HDface. The dataset is fully annotated for each semantic region (i.e. the time duration of each word that the human signer performs). Our proposed method achieves 92 .88 % accuracy for recognizing 100 ASL words in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on the Chalearn IsoGD dataset and achieves 76% accuracy which is 5.51% higher than the state-of-the-art work

show abstract

Section: Discussionmentioning

confidence: 99%

Recognizing American Sign Language Manual Signs from RGB-D Videos

Jing¹,

Huenerfauth²,

Tian³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Sensors are one of the most fundamental things in developing a virtual system. One of these systems is described in a research article by Liu, Fan, Li, and Zhang [9]. Using the particle filter model, they developed a hand tracking and gesture-recognition system that has a high tracking accuracy.…”

Section: Related Workmentioning

confidence: 99%

Creating an Interactive Musical Experience for a Concert Hall

Antoshchuk¹,

Kovalenko²,

Sieck³

2018

IJC

View full text Add to dashboard Cite

In this paper we introduce a Kinect based posture recognition approach that can classify the user’s pose and gesture and match them to a set of predefined musical instruments. The efficiency of the approach is then demonstrated using two applications. The Virtual Orchestra system uses pose- and gesture-recognition along with Augmented Reality technology to add a virtual musical instrument into the scene, both visually and audibly: the visual representation of the instrument is placed into the user’s hands and the sound of the corresponding instrument is played. An additional functionality is that the user can control the intensity and the pitch of the sound by changing the speed of his hand or finger movements. The Magic Mirror game is the second application developed for the Berlin Concert Hall that uses the posture recognition approach to introduce the visitors to some classical music pieces and familiarize them with various classical musical instruments.

show abstract

“…There are over 300 different sign languages in use worldwide [2]. Different sign languages use various sign language alphabets.…”

Section: Introductionmentioning

confidence: 99%

Real-time recognition of American sign language using long-short term memory neural network and hand detection

Abdulhamied

Nasr

Abdulkader

2023

IJEECS

View full text Add to dashboard Cite

Sign language recognition is very important for deaf and mute people because it has many facilities for them, it converts hand gestures into text or speech. It also helps deaf and mute people to communicate and express mutual feelings. This paper's goal is to estimate sign language using action detection by predicting what action is being demonstrated at any given time without forcing the user to wear any external devices. We captured user signs with a webcam. For example; if we signed “thank you”, it will take the entire set of frames for that action to determine what sign is being demonstrated. The long short-term memory (LSTM) model is used to produce a real-time sign language detection and prediction flow. We also applied dropout layers for both training and testing dataset to handle overfitting in deep learning models which made a good improvement for the final result accuracy. We achieved a 99.35% accuracy after training and implementing the model which allows the deaf and mute communicate more easily with society.

show abstract

RGBD Video Based Human Hand Trajectory Tracking and Gesture Recognition System

Cited by 10 publications

References 26 publications

Recognizing American Sign Language Manual Signs from RGB-D Videos

Recognizing American Sign Language Manual Signs from RGB-D Videos

Creating an Interactive Musical Experience for a Concert Hall

Real-time recognition of American sign language using long-short term memory neural network and hand detection

Contact Info

Product

Resources

About