Dynamic Gesture Classification for Vietnamese Sign Language Recognition

Vo, Duc-Hoang; Huynh, Huu-Hung; Doan, Phuoc-Mien; Meunier, Jean-Guy

doi:10.14569/ijacsa.2017.080357

Cited by 10 publications

(10 citation statements)

References 19 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even although, it also has many challenges like complex background problems, variation in lighting condition, changes of skin color, and the properties and configurations of camera (such as variations of scale, translation, rotation, view and occlusion). In [2]- [4], Microsoft Kinect camera are used to capture RGB-D image instead of RGB image to improve the results. However, Microsoft Kinect camera is hard to integrated into real world applications.…”

Section: Related Workmentioning

confidence: 99%

Deep Learning for Vietnamese Sign Language Recognition in Video Sequence

Vo¹,

Pham²,

Nguyen³

2019

IJMLC

View full text Add to dashboard Cite

With most of Vietnamese hearing impaired individuals, Vietnamese Sign Language (VSL) is the only choice for communication. Thus, there are more and more study about the automatic translation of VSL to make a bridge between hearing impaired people and normal ones. However, automatic VSL recognition in video brings many challenges due to the orientation of camera, hand position and movement, inter hand relation, etc. In this paper, we present some feature extraction approaches for VSL recognition including spatial and scene-based features. Instead of relying on a static image, we specifically capture motion information between frames in a video sequence. For the recognition task, beside the traditional method of sign language recognition such as SVM, we additionally propose to use deep learning technique for VSL recognition for finding the dependence of each frame in video sequences. We collected two VSL datasets of the relative family topic (VSL-WRF) like father, mother, uncle, aunt.... The first one includes 12 words in Vietnamese language which only have a little change between frames. While the second one contains 15 with gestures involving the relative position of the body parts and orientation of the motion. Moreover, the data augmentation technique is proposed to gain more information of hand movement and hand position. The experiments achieved the satisfactory results with accuracy of 88.5% (traditional SVM) and 95.83% (deep learning). It indicates that deep learning combining with data augmentation technique provides more information about the orientation or movement of hand, and it would be able to improve the performance of VSL recognition system. Index Terms-Vietnamese sign language (VSL), VSL recognition, local descriptors, spatial feature, scene-based feature, Motion-based feature, deep learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Learning for Vietnamese Sign Language Recognition in Video Sequence

Vo¹,

Pham²,

Nguyen³

2019

IJMLC

View full text Add to dashboard Cite

show abstract

“…6, December 2019 independent equipment, static and/or dynamic hand gesture processing algorithm, etc. This problem has been researched for a long time because of its many practical applications such as sign language [8], [16], entertainment [3], and human-machine interaction [1], [2]. As hand gesture recognition is seen by information flows such as RGB image [5], [8], [17], Depth image [4], [6], optical flow [17], and so on.…”

Section: Related Workmentioning

confidence: 99%

“…Their system requires a high configuration computer, and RGB and Depth multi-views dataset is captured by two camera but this dataset does not designed to evaluate the impact of viewpoints. In [2], [16], RGB-D images from the Kinect sensor are used to improve the hand gesture accuracy results. However, a real application is hard to deploy and integrate.…”

Section: Related Workmentioning

confidence: 99%

“…On the first hand, we utilize ISOMAP algorithm [9] to present hand shapes of dynamic hand gesture. On the other hand, the hand trajectories are reconstructed using a conventional KLT trackers [10], [16] as performed in [5]. Each hand posture Pi in a dynamic hand gesture is normalized and composed by two hand trajectory elements (Tri = [xi, yi]) and three hand shape elements (Yi=[Yi,1, Yi,2,Yi,3]) as equation 1following:…”

Section: B Hand Gesture Extraction With Rgb Modalitymentioning

confidence: 99%

See 1 more Smart Citation

Improving Dynamic Hand Gesture Recognition on Multi-views with Multi-modalities

Doan¹,

Nguyen²

2019

IJMLC

View full text Add to dashboard Cite

Hand gesture recognition topic has been researched for many recent decades because it could be used in many fields as sign language, virtual game, human-robot interaction, entertainment and so on. However, this problem has been faced to many challenges such as combination of multi information in a temporal flow in order to understand the meaning of human hand gesture. In recent times, thanks to the advances in hardware technologies such as readily available 3D cameras, Kinect sensors, and etc. The impressive performance of cutting-edge techniques in computer vision, which is known as: manifold learning, deep learning techniques and/or the presentation of various multimodal fusion strategies. There have been many improvements in exploiting of features from multimodal data to effectively solve human hand gesture recognition tasks. Therefore, this paper focuses on solving the problem of dynamic hand gesture recognition in our daily life. We consider methods for extracting features of different data sources (RGB images and depth images) based on both manifold learning and deep learning technique. For RGB information, a manifold technique is performed to extract spatial feature that is then composed with temporal feature extracted by KLT technique. Among many deep learning architectures proposed in the literature that achieved good results in detecting human activities, I studied and proposed a simple convolutional neural network to extract feature of depth motion map. This technique extracts hand features from depth information which combines spatial and temporal aspects. Besides that, fusion algorithms are deployed to unite with those extracted features and enhance the accuracy of a final dynamic hand gesture results. Evaluation results confirm that the best accuracy rate achieves at 84.7% that is significantly higher than results from previous works (at 78.4%). The proposed method suggests a feasible solution addressing technical issues in using multimodality and multi-viewpoint of hand gestures.

show abstract

“…The user is supposed to stay close to the camera and occupies a significant portion of the camera's field of view [12]. Microsoft's Kinect is a depth camera that acquires gestures with rich depth information and has demonstrated a notable performance of light sensitivity in gesture recognition [18]. However, there are still unresolved issues regarding gesture finding and segmenting with Kinect sensors [10].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Hand Gesture to Text using Leap Motion

Jamaludin¹,

Fang²

2019

IJACSA

View full text Add to dashboard Cite

This paper presents a prototype for converting dynamic hand gestures to text by using a device called Leap Motion. It is one of the motion tracking technologies, which could be used for recognising hand gestures without the need of wearing any external devices or capturing any images and videos. In this study, five custom dynamic hand gestures of American Sign Language were created with Leap Motion to measure the recognition accuracy of the proposed prototype using the Geometric Template Matching, Artificial Neural Network, and Cross-Correlation algorithms. The experimental results showed that the prototype achieved recognition accuracy of more than 90% in the training phase and about 60% in the testing phase.

show abstract

Dynamic Gesture Classification for Vietnamese Sign Language Recognition

Cited by 10 publications

References 19 publications

Deep Learning for Vietnamese Sign Language Recognition in Video Sequence

Deep Learning for Vietnamese Sign Language Recognition in Video Sequence

Improving Dynamic Hand Gesture Recognition on Multi-views with Multi-modalities

Dynamic Hand Gesture to Text using Leap Motion

Contact Info

Product

Resources

About